Go by Example: Worker Pools

Go 1.23

Implement the Worker Pool pattern to manage concurrency and resources efficiently. This example shows how to dispatch jobs to a fixed number of worker goroutines using channels, preventing resource exhaustion and maximizing throughput.

Code

package main

import (
    "fmt"
    "time"
)

// worker processes jobs from the jobs channel and sends results to results channel
func worker(id int, jobs <-chan int, results chan<- int) {
    for j := range jobs {
        fmt.Printf("worker %d started job %d\n", id, j)
        // Simulate work time
        time.Sleep(time.Second)
        fmt.Printf("worker %d finished job %d\n", id, j)
        results <- j * 2
    }
}

func main() {
    const numJobs = 5
    const numWorkers = 3

    // Buffered channels for jobs and results
    jobs := make(chan int, numJobs)
    results := make(chan int, numJobs)

    // Start 3 workers
    // They are initially blocked because there are no jobs yet
    for w := 1; w <= numWorkers; w++ {
        go worker(w, jobs, results)
    }

    // Send 5 jobs to the workers
    for j := 1; j <= numJobs; j++ {
        jobs <- j
    }
    // Close jobs channel to signal no more work
    close(jobs)

    // Collect all results
    // This ensures we wait for all jobs to complete
    for a := 1; a <= numJobs; a++ {
        <-results
    }
    
    fmt.Println("All jobs processed")
}

Explanation

The Worker Pool pattern is a common concurrency pattern used to limit the number of active goroutines while processing a stream of tasks. This prevents resource exhaustion when dealing with thousands of tasks.

In this example, we create a fixed number of worker goroutines (3) to process a larger number of jobs (5). The workers consume tasks from a shared "jobs" channel and send output to a "results" channel.

Benefits of Worker Pools:

Controls resource usage (CPU, memory, network connections)
Reuses goroutines instead of creating/destroying them constantly
Provides backpressure if the job queue fills up

Code Breakdown

9

The worker function takes a receive-only channel for jobs (<-chan) and a send-only channel for results (chan<-). This type safety prevents workers from accidentally sending jobs or reading results.

10

Iterating over a channel with range continues until the channel is closed and empty. This allows workers to automatically stop when we close the jobs channel.

38

Closing the jobs channel is the signal to all workers that no more tasks are coming. Once they finish their current task, their range loop will terminate.

Samplebadu

Go by Example: Worker Pools

Code

Explanation

Code Breakdown