I was deep in a 2 am incident when the heap usage spiked from 300 MiB to 2 GiB in under a minute. The culprit? A single health‑check endpoint that spun up a goroutine per request and never let it die. The container crashed, the load‑balancer kept retrying, and the outage lasted 12 minutes. It took us three post‑mortems to figure out that the “leak” was a *goroutine* masquerading as a memory leak.
- Never let a request‑scoped goroutine outlive its
context.Context. - Configure
http.Servertimeouts (ReadHeader, Idle, Write) and always callShutdown. - Use
errgrouporrun.Groupfor linked lifecycles. - Detect leaks early with
uber-go/goleakand runtime metrics. - When in doubt, bound concurrency with a worker pool.
Before you start: Go 1.24 (or newer), familiarity with context, net/http, and sync.WaitGroup. Optional: otel SDK, Prometheus client_golang, and pprof enabled.
Prevent goroutine leaks in Go HTTP servers
Prevent goroutine leaks in Go HTTP servers by rigorously propagating and respecting `context.Context` for cancellation, configuring `http.Server` timeouts (ReadHeader, Idle), and implementing graceful shutdown with `Shutdown()`. Use tools like `errgroup` for structured concurrency and `goleak` in tests to detect leaks early.
Introduction: Why Goroutine Leaks Cripple Production Go Servers
The Hidden Multiplier Effect of Leaked Goroutines
A goroutine is light—~2 KB stack, a few hundred bytes of bookkeeping. That sounds trivial until you multiply it by thousands. In our incident the leaked handler left **3 k** goroutines per second hanging in `net/http.(*conn).serve`. After 30 seconds the runtime held ~90 k stacks, each consuming memory and CPU for context switches. The scheduler had to shuffle them, causing latency spikes even before the container OOM‑killed.
How a Simple HTTP Handler Can Consume Gigabytes
func health(w http.ResponseWriter, r *http.Request) {
go func() {
// Simulate work that never sees r.Context().Done()
time.Sleep(10 * time.Minute)
}()
w.WriteHeader(http.StatusOK)
}
No error handling, no cancellation, just fire‑and‑forget. Under 10 k RPS the process hits 2 GiB of heap in minutes. The docs show the snippet as “quick‑and‑dirty”, but in production you cannot afford that hidden multiplier.
**My take:** The classic “goroutine‑per‑request” model is fine **only** when every spawned goroutine is guaranteed to exit when the request does. Anything else is a ticking time bomb.
Goroutine Leak Fundamentals: Contexts, Channels, and Blocking
The Role of `context.Context` in Resource Lifecycle
`context.Context` is the only built‑in way to propagate cancellation downstream. When a client disconnects, the server cancels the request’s root context. Any child goroutine that respects `<-ctx.Done()` exits cleanly. The pattern looks like:
func doWork(ctx context.Context, id int) error {
select {
case <-time.After(5 * time.Second):
// work finished
case <-ctx.Done():
return ctx.Err()
}
return nil
}
If you ignore the `Done` channel, the goroutine lives forever.
Blocking Operations That Trap Goroutines Forever
- **Channel receives without timeout** – `val := <-ch` can block forever if the sender stops.
- **I/O reads** – `net.Conn.Read` blocks until data or error; without a deadline it never returns on a dead client.
- **Database calls** – Many drivers honour the context, but some (e.g., old MySQL drivers) ignore it, leaving the goroutine hung.
Instrumentation: Using `runtime.NumGoroutine` and `pprof`
log.Printf("goroutines: %d", runtime.NumGoroutine())
Add a Prometheus gauge:
var goroutineGauge = prometheus.NewGauge(prometheus.GaugeOpts{
Name: "go_goroutine_count",
Help: "Current number of goroutines.",
})
prometheus.MustRegister(goroutineGauge)
go func() {
for {
goroutineGauge.Set(float64(runtime.NumGoroutine()))
time.Sleep(10 * time.Second)
}
}()
For deep inspection, enable `net/http/pprof` and visit `/debug/pprof/goroutine?debug=2`. Look for stacks that end in `<-chan struct {}>` or a blocking `select` without a `<-ctx.Done()>` clause.
**Tip:** See our [detailed tutorial on interpreting pprof heap/goroutine profiles](/debugging-go-pprof-profiles/) for a step‑by‑step walkthrough.
Critical Technique 1: Context Propagation & Cancellation for HTTP Handlers
Correctly Wiring `request.Context()` Through Your Call Chain
Never pass `*http.Request` as the sole carrier of cancellation. Extract the context early and pass it explicitly:
func handler(w http.ResponseWriter, r *http.Request) {
ctx := r.Context()
if err := processRequest(ctx, r); err != nil {
http.Error(w, err.Error(), http.StatusInternalServerError)
return
}
w.WriteHeader(http.StatusOK)
}
Every downstream function must accept `ctx context.Context`. Libraries that don’t expose a context parameter (e.g., some third‑party loggers) should be wrapped.
The Defer `context.CancelFunc()` Pattern You Must Use
When you create a derived context, defer its cancel *immediately* to avoid leaks:
func processRequest(parent context.Context, r *http.Request) error {
ctx, cancel := context.WithTimeout(parent, 2*time.Second)
defer cancel() // guarantees resources free even on early return
// pass ctx further...
return doWork(ctx, 42)
}
A common mistake is to place `defer cancel()` after some long‑running code; if that code never returns, the cancel never runs.
Custom Timeout Contexts for Long‑Running Handler Logic
For endpoints that do heavy aggregation, a per‑request timeout protects the whole chain:
func aggregateHandler(w http.ResponseWriter, r *http.Request) {
ctx, cancel := context.WithTimeout(r.Context(), 5*time.Second)
defer cancel()
data, err := fetchAllSources(ctx)
if err != nil {
if errors.Is(err, context.DeadlineExceeded) {
http.Error(w, "timeout", http.StatusGatewayTimeout)
} else {
http.Error(w, "internal error", http.StatusInternalServerError)
}
return
}
json.NewEncoder(w).Encode(data)
}
Critical Technique 2: HTTP Server and Transport Configuration for Safety
Setting ReadHeader, Read, Idle, and Write Timeouts on `http.Server`
srv := &http.Server{
Addr: ":8080",
Handler: mux,
ReadHeaderTimeout: 5 * time.Second, // most important
ReadTimeout: 10 * time.Second,
WriteTimeout: 10 * time.Second,
IdleTimeout: 30 * time.Second,
MaxHeaderBytes: 1 << 20, // 1 MiB
}
*ReadHeaderTimeout* stops a slow‑loris client that drags the header line for minutes. In 2026, the Go runtime enforces this timeout more strictly than before, making it a non‑negotiable guard.
Why `http.Server.Shutdown` Is Non‑Negotiable for Graceful Exit
Calling `srv.Shutdown(ctx)` stops new connections, then waits for existing handlers to finish until `ctx` expires. Skipping this step leaves half‑closed connections, and the OS may keep sockets open, trapping goroutines forever.
go func() {
if err := srv.ListenAndServe(); err != nil && err != http.ErrServerClosed {
log.Fatalf("listen: %v", err)
}
}()
// Wait for SIGINT/SIGTERM …
<-stopCh
shutdownCtx, cancel := context.WithTimeout(context.Background(), 10*time.Second)
defer cancel()
if err := srv.Shutdown(shutdownCtx); err != nil {
log.Printf("forced shutdown: %v", err)
}
Configuring `http.Transport` MaxIdleConns and IdleConnTimeout
For outbound calls (client side), the default transport keeps idle connections forever. That can lock goroutines inside the transport’s connection reaper.
transport := &http.Transport{
MaxIdleConns: 100,
IdleConnTimeout: 30 * time.Second,
DisableKeepAlives: false,
TLSHandshakeTimeout: 5 * time.Second,
}
client := &http.Client{
Transport: transport,
Timeout: 15 * time.Second, // end‑to‑end safety net
}
Critical Technique 3: Graceful Shutdown with Worker Pool Drainage
Implementing a Shutdown Signal Channel (`os.Signal`)
stopCh := make(chan os.Signal, 1)
signal.Notify(stopCh, os.Interrupt, syscall.SIGTERM)
go func() {
<-stopCh
// start graceful shutdown logic
}()
Draining Background Worker Goroutines with `sync.WaitGroup`
var wg sync.WaitGroup
poolSize := 20
jobs := make(chan job, 100)
for i := 0; i < poolSize; i++ {
wg.Add(1)
go func(id int) {
defer wg.Done()
for j := range jobs {
if err := handleJob(j); err != nil {
log.Printf("worker %d error: %v", id, err)
}
}
}(i)
}
// On shutdown:
close(jobs) // signal workers to stop receiving
wg.Wait() // blocks until all workers finish
Timeout Wrappers for Shutdown to Prevent Deadlock on Stuck Goroutines
Even with a `WaitGroup`, a single worker could block forever. Wrap the wait in a timeout:
shutdownDone := make(chan struct{})
go func() {
wg.Wait()
close(shutdownDone)
}()
select {
case <-shutdownDone:
log.Println("all workers drained")
case <-time.After(8 * time.Second):
log.Println("force exit: workers stuck")
}
This two‑stage approach (graceful then hard) addresses the production gotcha where a rogue background job refuses to respect cancellation.
Critical Technique 4: Structured Concurrency with `errgroup` and `run.Group`
Leveraging `golang.org/x/sync/errgroup` for Linked Goroutine Lifecycles
g, ctx := errgroup.WithContext(r.Context())
g.Go(func() error { return fetchFromA(ctx) })
g.Go(func() error { return fetchFromB(ctx) })
g.Go(func() error { return fetchFromC(ctx) })
if err := g.Wait(); err != nil {
// one of the fetches failed or context canceled
log.Printf("aggregate error: %v", err)
}
All child goroutines inherit the same cancellation and error propagation. If any returns non‑nil, the group cancels the shared context, preventing leaks.
The `run.Group` Pattern for Explicit Startup/Shutdown Ordering
`run.Group` (from `github.com/tombuilders/RunGroup`) gives you a pair of functions: one to start, one to stop.
var g run.Group
// HTTP server
{
srv := &http.Server{Addr: ":8080", Handler: mux}
g.Add(func() error {
return srv.ListenAndServe()
}, func(err error) {
ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
defer cancel()
srv.Shutdown(ctx)
})
}
// Background worker
{
ctx, cancel := context.WithCancel(context.Background())
g.Add(func() error {
return runWorker(ctx)
}, func(err error) {
cancel()
})
}
// Run everything
if err := g.Run(); err != nil && !errors.Is(err, http.ErrServerClosed) {
log.Fatalf("run group error: %v", err)
}
The stop function is guaranteed to run even if the start function returns early, keeping the lifecycle tidy.
Propagating Errors and Cancellation Through the Goroutine Tree
Never swallow `ctx.Err()`. Always surface it up the call stack so the parent can decide to cancel siblings.
if err := doSomething(ctx); err != nil {
if errors.Is(err, context.Canceled) {
return err // let errgroup cancel others
}
// handle other errors
}
Real‑World Case Study & Production Gotchas
Analyzing a Leak from a Misconfigured Database Connection Pool
Our fintech platform used `pgxpool` with `MaxConns=0` (unlimited). Under load the pool opened 12 k connections, each spawning a goroutine for health‑check pings. The goroutine count exploded, and the OOM alarm fired. Limiting the pool and adding a per‑request context to the query resolved it.
poolConfig, _ := pgxpool.ParseConfig(os.Getenv("DATABASE_URL"))
poolConfig.MaxConns = 100
pool, _ := pgxpool.NewWithConfig(context.Background(), poolConfig)
The “Slow Client” Attack: How Idle Timeouts Prevent It
A malicious client opened a connection and sent headers one byte every 30 seconds. Without `ReadHeaderTimeout` the server kept the connection alive, holding a goroutine per pending request. Adding a 5‑second header timeout closed the socket immediately.
Integrating with OpenTelemetry for Goroutine Count Metrics
meter := otel.GetMeterProvider().Meter("myapp")
goroutineObs, _ := meter.Int64ObservableGauge("go_goroutine_count")
meter.RegisterCallback(func(ctx context.Context, o metric.Observer) error {
o.ObserveInt64(goroutineObs, int64(runtime.NumGoroutine()))
return nil
}, goroutineObs)
These metrics show up in Grafana dashboards, letting you spot abnormal spikes before they become fatal.
Advanced Patterns and Architectural Trade‑offs
| Pattern | When to Use | Pros | Cons |
|---|---|---|---|
| Goroutine‑per‑request | Low‑traffic, simple CRUD | Simple mental model | Unbounded growth under load |
| Bounded worker pool | High RPS, expensive I/O or CPU work | Predictable concurrency, memory cap | Slight latency due to queueing |
| `errgroup` + timeout context | Dependent parallel tasks (e.g., fan‑out) | Automatic cancellation, error surfacing | Must pass context everywhere |