Concurrency Patterns in Go: Fan-out, Fan-in, and Worker Pools

Concurrency is a key feature of the Go programming language, and Go provides various concurrency patterns to efficiently manage parallel tasks. In this section, we’ll explore three common concurrency patterns in Go: the fan-out pattern, the fan-in pattern, and the use of worker pools.

The Fan-Out Pattern

The fan-out pattern is a concurrency pattern used when you want to distribute a task to multiple goroutines for parallel execution. This pattern is useful when you have a time-consuming task that can be divided into smaller subtasks. Here’s how it works:

1. Create a Task Channel: First, create a channel to send subtasks to worker goroutines.
2. Start Worker Goroutines: Start multiple worker goroutines that listen to the task channel.
3. Send Tasks: Send subtasks to the task channel from the main goroutine.
4. Wait for Results: Collect and aggregate results from worker goroutines.

Example: Fan-Out Pattern

Let’s consider an example where you want to calculate the square of a list of numbers concurrently. Here’s a simple implementation using the fan-out pattern:

package main

import (
    "fmt"
    "sync"
)

func main() {
    numbers := []int{1, 2, 3, 4, 5}
    taskChan := make(chan int)
    var wg sync.WaitGroup

    // Start worker goroutines
    for i := 0; i < 3; i++ {
        wg.Add(1)
        go func() {
            defer wg.Done()
            for n := range taskChan {
                result := n * n
                fmt.Println("Square of", n, "is", result)
            }
        }()
    }

    // Send tasks to worker goroutines
    for _, num := range numbers {
        taskChan <- num
    }

    close(taskChan)
    wg.Wait()
}

In this example, we create a channel “taskChan” to send subtasks to worker goroutines. We start three worker goroutines to process the tasks in parallel. The main goroutine sends numbers to be squared to the task channel, and workers calculate and print the squares.

The Fan-In Pattern

The fan-in pattern is used to combine and aggregate results from multiple goroutines into a single channel. This is useful when you have multiple goroutines producing data, and you want to collect and process that data in one place. Here’s how it works:

1. Create Worker Goroutines: Start multiple worker goroutines that produce data and send it to their own channels.
2. Create a Fan-In Function: Create a function that combines data from multiple channels into a single channel.
3. Use the Combined Channel: Read data from the combined channel and process it.

Example: Fan-In Pattern

Let’s say you have two goroutines that generate even and odd numbers and you want to combine them into a single channel. Here’s an example using the fan-in pattern:

package main

import "fmt"

func main() {
    evenChan := make(chan int)
    oddChan := make(chan int)
    combinedChan := make(chan int)

    // Even numbers generator
    go func() {
        for i := 0; i <= 10; i += 2 {
            evenChan <- i
        }
        close(evenChan)
    }()

    // Odd numbers generator
    go func() {
        for i := 1; i <= 10; i += 2 {
            oddChan <- i
        }
        close(oddChan)
    }()

    // Fan-in function
    go func() {
        for {
            select {
            case num, ok := <-evenChan:
                if ok {
                    combinedChan <- num
                }
            case num, ok := <-oddChan:
                if ok {
                    combinedChan <- num
                }
            }
        }
        close(combinedChan)
    }()

    // Read from the combined channel
    for num := range combinedChan {
        fmt.Println("Received:", num)
    }
}

In this example, we have two goroutines generating even and odd numbers. The fan-in function combines the results from both channels into a single channel, which is then processed in the main goroutine.

Worker Pools

Worker pools are a way to manage concurrent execution of tasks with a limited number of worker goroutines. This pattern is useful when you want to control the number of simultaneous tasks running in parallel. Here’s how it works:

1. Create a Task Queue: Set up a channel to receive tasks that need to be executed.
2. Start Worker Goroutines: Launch a fixed number of worker goroutines that listen to the task queue.
3. Send Tasks: Send tasks to the task queue, and worker goroutines pick up and execute them.

Example: Worker Pool

Let’s consider an example where you have a list of tasks that need to be processed concurrently by a limited number of workers:

package main

import (
    "fmt"
    "sync"
)

func main() {
    tasks := []string{"Task 1", "Task 2", "Task 3", "Task 4", "Task 5"}
    taskChan := make(chan string, 2) // Limited to 2 tasks at a time
    var wg sync.WaitGroup

    // Start worker goroutines
    for i := 0; i < 2; i++ {
        wg.Add(1)
        go func() {
            defer wg.Done()
            for task := range taskChan {
                fmt.Println("Processing", task)
            }
        }()
    }

    // Send tasks to the task queue
    for _, task := range tasks {
        taskChan <- task
    }

    close(taskChan)
    wg.Wait()
}

In this example, we create a task channel with a capacity of 2, limiting the number of concurrent tasks to 2. Worker goroutines pick up tasks from the channel and process them.

Conclusion

Concurrency patterns in Go, such as the fan-out, fan-in, and worker pools, provide powerful tools for managing parallel tasks efficiently. These patterns help you make the most of Go’s concurrency features and ensure that your applications can handle tasks concurrently and in a controlled manner.