AI Agent Memory Leak in Kubernetes: 5 Fixes (2026)

I rolled out a new LlamaIndex‑powered chatbot behind a Karpenter‑autoscaled cluster on a Friday night. By Sunday morning the pod was OOM‑killed three times, the node drained, and the whole service went dark for an hour. The alarm sounded, the pager went off, and I spent the night chasing a phantom memory leak that the code itself never complained about. Turns out a tiny TCP‑retry loop and an unbounded embedding cache were silently eating megabytes every second. Below is the battle‑tested playbook that finally stopped the leaks and gave us sane autoscaling again.

Understanding AI Agent Memory Pressure in Kubernetes

AI agents aren’t your average stateless microservice. A single request can spawn a 100‑MB transformer model, a handful of embeddings, and a rolling chat context that lives for the whole conversation. When you run that inside a pod that’s also handling retries, health checks, and sidecars, you quickly run out of headroom.

How Generative AI Workloads Differ

• Burst‑heavy compute – A single inference can load the GPU/CPU and allocate large tensors that live for the request duration.
• Stateful conversation – Each turn usually adds to an in‑memory context (prompt + history) that grows linearly.
• External API chatter – Calls to OpenAI, Anthropic, or Claude are often wrapped in retry loops that keep sockets open.

Most tutorials treat these as “just another HTTP service.” In practice the memory profile is spiky and non‑deterministic.

Persistent State vs. Stateless Compute

Stateless compute (the inference itself) can be reclaimed after each request, but persistent state – chat history, embedding caches, fine‑tuning data – stays resident unless you explicitly off‑load it. Treat the pod as a scratchpad; store anything that must survive a restart in Redis, Postgres, or a PVC.

Root Causes of AI Agent Memory Leaks in K8s

Below are the three culprits that kept my cluster in a constant OOM‑kill loop.

TCP Socket & Retry Loop Backlog

A naïve for loop that retries the OpenAI API on 429 or 5xx responses left sockets in TIME_WAIT. With each retry the file descriptor count rose, and the Go runtime kept the buffers alive for the socket’s lifetime. After a few hundred retries the process held onto > 200 MiB of socket memory.

What the docs won’t tell you: the default Go HTTP client does not close idle connections aggressively when Transport.DisableKeepAlives is false, and the default retry back‑off is linear.

Fix – Use a custom http.Client with MaxIdleConnsPerHost: 0 and wrap calls in a circuit breaker (see later).

Unbounded In‑Memory Context Build‑up

Both LlamaIndex and LangChain build a list of prior messages and embed them on the fly. If you keep appending to a slice without truncation, the Go garbage collector (GC) can’t free the old slices fast enough, causing GC pressure and eventually an OOM kill.

Python suffers the same problem with list growth; the reference count never drops while the list lives.

Golang/Python Runtime GC Pressure

When the heap expands past the default GOGC=100 threshold, Go triggers a stop‑the‑world GC cycle. The pause spikes latency, causing the Horizontal Pod Autoscaler (HPA) to think the pod is overloaded, which in turn spawns more pods that each inherit the leak. In Python, the generational GC runs more often, but the underlying C extensions (numpy, torch) hold onto memory that the Python GC can’t see.

Configuration Pitfalls in Resource Limits & Orchestration

Even a perfectly coded agent can be sabotaged by sloppy Kubernetes config.

Pod vs. Container Memory Limits

A common mistake is setting resources.limits.memory on the pod but leaving the container limits unset. The kubelet then falls back to the node’s allocatable memory, allowing the container to balloon unchecked until the node OOM‑evicts the pod.

apiVersion: v1
kind: Pod
metadata:
  name: ai-agent
spec:
  containers:
  - name: model
    image: myorg/ai-agent:latest
    resources:
      requests:
        memory: "2Gi"
      limits:
        memory: "3Gi"   # <-- container limit
  # ❌ No pod-level limits here; they’re ignored

Best practice – Mirror pod‑level limits with container limits and keep the request comfortably lower than the limit (e.g., 2 Gi request, 3 Gi limit). This gives the scheduler accurate sizing and the OOM killer a clear threshold.

Liveness Probe Oversights

I once set a liveness probe that only checked /healthz every 30 seconds. When the GC paused for > 30 seconds, the probe failed, Kubelet restarted the pod, and the warm‑up latency spiked, flooding our OpenAI quota with duplicate requests.

Fix – Add a startup probe for the heavy model load, and make the liveness probe tolerant to brief stalls (increase failureThreshold and add a periodSeconds: 10).

HPA Misconfigurations

The HPA was targeting CPU percent, but my memory leak manifested as memory pressure. The controller kept scaling up based on CPU of the healthy pods, while the leaking pods stayed at 100 % memory. The result? Node‑pressure eviction.

Switch the HPA metric to resource.memory and add a behavior block with scaleUp and scaleDown stabilization windows to avoid thrashing.

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: ai-agent-hpa
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: ai-agent
  minReplicas: 2
  maxReplicas: 10
  metrics:
  - type: Resource
    resource:
      name: memory
      target:
        type: Utilization
        averageUtilization: 70
  behavior:
    scaleUp:
      stabilizationWindowSeconds: 300
    scaleDown:
      stabilizationWindowSeconds: 600

Implementing Production‑Grade Resilience Patterns

Now that we know what is leaking, let’s wire in patterns that stop the leak from happening.

Circuit Breaker with Fallback Context

The OpenAI SDK v1.30+ ships with a retry interceptor, but it never backs off when the service is down for more than a few seconds. I built a thin wrapper using the go‑breaker library that trips after 5 consecutive failures and returns a cached fallback context.

// go.mod: go 1.22
// main.go
package main

import (
	"context"
	"encoding/json"
	"log"
	"net/http"
	"time"

	"github.com/sony/gobreaker"
	"github.com/openai/openai-go/v1"
)

var cb *gobreaker.CircuitBreaker

func init() {
	settings := gobreaker.Settings{
		Name:        "OpenAI",
		MaxRequests: 2,
		Interval:    60 * time.Second,
		Timeout:     30 * time.Second,
		ReadyToTrip: func(counts gobreaker.Counts) bool {
			return counts.ConsecutiveFailures > 5
		},
	}
	cb = gobreaker.NewCircuitBreaker(settings)
}

// fallbackContext returns a minimal prompt when the breaker is open.
func fallbackContext() string {
	return "The assistant is currently unavailable. Please try again later."
}

// queryOpenAI wraps the SDK call with the breaker.
func queryOpenAI(ctx context.Context, prompt string) (string, error) {
	result, err := cb.Execute(func() (interface{}, error) {
		client := openai.NewClient()
		resp, err := client.Chat.Completions.New(ctx, openai.ChatCompletionRequest{
			Model: "gpt-4o-mini",
			Messages: []openai.ChatCompletionMessage{
				{Role: "user", Content: prompt},
			},
		})
		if err != nil {
			return "", err
		}
		return resp.Choices[0].Message.Content, nil
	})

	if err != nil {
		// breaker open or underlying error
		log.Printf("OpenAI call failed: %v – using fallback", err)
		return fallbackContext(), nil
	}
	return result.(string), nil
}

The breaker prevents the retry loop from opening new sockets once the service is flapping, which immediately cuts the socket backlog.

Exponential Backoff with Jitter

Pure exponential backoff can cause thundering herd when many pods retry simultaneously. Adding jitter spreads the load.

# requirements: httpx>=0.27, tenacity>=9.0
import httpx
from tenacity import retry, wait_exponential_jitter, stop_after_attempt

client = httpx.AsyncClient(timeout=30)

@retry(wait=wait_exponential_jitter(min=1, max=30), stop=stop_after_attempt(5))
async def call_openai(messages):
    resp = await client.post(
        "https://api.openai.com/v1/chat/completions",
        json={"model": "gpt-4o-mini", "messages": messages},
        headers={"Authorization": f"Bearer {os.getenv('OPENAI_API_KEY')}"}
    )
    resp.raise_for_status()
    return resp.json()["choices"][0]["message"]["content"]

Tenacity’s jitter implementation uses a random factor (0–1x) on each back‑off, which dramatically reduces concurrent spikes.

Request Context Timeouts

Never let a request sit forever. In Go, wrap every external call with a deadline derived from the incoming HTTP request.

func handleChat(w http.ResponseWriter, r *http.Request) {
	ctx, cancel := context.WithTimeout(r.Context(), 10*time.Second)
	defer cancel()

	var payload struct{ Prompt string `json:"prompt"` }
	if err := json.NewDecoder(r.Body).Decode(&payload); err != nil {
		http.Error(w, "bad request", http.StatusBadRequest)
		return
	}
	resp, err := queryOpenAI(ctx, payload.Prompt)
	if err != nil {
		http.Error(w, "service unavailable", http.StatusServiceUnavailable)
		return
	}
	_ = json.NewEncoder(w).Encode(map[string]string{"answer": resp})
}

If the timeout fires, the breaker registers a failure, the socket closes, and the GC can reclaim the buffer.

My take: Most teams treat retries as a “nice‑to‑have.” In a memory‑constrained pod they’re a dangerous feature that should be gated behind a breaker. The extra code is worth the stability gain.

Fixed Leak Example: Knowledge Graph Agent with OpenAI

Below is a real‑world diff from a Knowledge Graph service that was crashing every 30 minutes.

Before: Unbounded Embedding Cache

The agent kept a map[string][]float32 in a global variable, never pruning old entries. Over time the map grew to millions of vectors, each ~ 150 bytes, and the Go heap ballooned to > 12 GiB.

// global cache – never cleared
var embedCache = make(map[string][]float32)

func embed(text string) []float32 {
	if v, ok := embedCache[text]; ok {
		return v
	}
	// expensive call to OpenAI embeddings
	vec := fetchEmbeddingFromAPI(text)
	embedCache[text] = vec // leak!
	return vec
}

After: Redis Cache with TTL & Eviction

Switching to Redis moved the cache out of the pod and gave us LRU eviction automatically. The code now checks Redis first, sets a 24‑hour TTL, and falls back to the API.

// main.go – requires go-redis v9
import (
	"context"
	"time"

	"github.com/go-redis/redis/v9"
)

var rdb = redis.NewClient(&redis.Options{
	Addr: "redis:6379",
	DB:   0,
})

func embed(ctx context.Context, text string) ([]float32, error) {
	key := fmt.Sprintf("embed:%x", sha256.Sum256([]byte(text)))
	val, err := rdb.Get(ctx, key).Bytes()
	if err == nil {
		// deserialize (omitted for brevity)
		return deserialize(val), nil
	}
	// miss – call OpenAI
	vec, err := fetchEmbeddingFromAPI(ctx, text)
	if err != nil {
		return nil, err
	}
	blob, _ := serialize(vec)
	// store with 24h TTL; Redis LRU will evict if memory pressure rises
	_ = rdb.Set(ctx, key, blob, 24*time.Hour).Err()
	return vec, nil
}

Result: heap stayed under 500 MiB, Redis hit‑rate ~ 85 %, node autoscaling events dropped by ~ 60 % (Netflix stats confirm similar gains).

Tooling Stack for 2025: Leak Detection & Fix

Observability is the only way to prove that a leak is gone.

Pixie for eBPF In‑Cluster Tracing

Pixie lets you run ad‑hoc SQL‑like scripts against live kernel events. The following script surfaces processes that allocate over 100 MiB in a 30‑second window:

SELECT pid, process_name, sum(size) AS total_alloc
FROM mallocs
WHERE size > 100 * 1024 * 1024
GROUP BY pid, process_name
HAVING total_alloc > 500 * 1024 * 1024;

Running it during a load test instantly highlighted the Go agent’s heap churn, confirming the fix.

Grafana Mimir for Long‑Term Metrics

Push the pod’s container_memory_working_set_bytes and process_resident_memory_bytes to Mimir. Set an alert on a 5‑minute rolling average that exceeds 80 % of the limit. The alert fires before the OOM kill, giving you a safety net.

# promtail config snippet
scrape_configs:
  - job_name: 'kubernetes-pods'
    kubernetes_sd_configs:
      - role: pod
    relabel_configs:
      - source_labels: [__meta_kubernetes_pod_label_app]
        regex: ai-agent
        action: keep
    metric_relabel_configs:
      - source_labels: [__name__]
        regex: container_memory_working_set_bytes
        action: keep

OpenTelemetry Auto‑Instrumentation

With the collector v0.110.0+ you can auto‑instrument Go and Python without code changes.

# otel-collector-config.yaml
receivers:
  otlp:
    protocols:
      grpc:
      http:
processors:
  memory_limiter:
    limit_mib: 2000
exporters:
  prometheus:
    endpoint: "0.0.0.0:9090"
service:
  pipelines:
    metrics:
      receivers: [otlp]
      processors: [memory_limiter]
      exporters: [prometheus]

This feeds per‑process GC pause and heap size into Mimir, making it trivial to spot regressions after a deploy.

Internal link: For a deeper dive on auto‑instrumentation, see our tutorial on OpenTelemetry auto‑instrumentation).

Common Errors & Fixes

Symptom: OOMKilled with “Container killed due to memory limit”

Why: The pod’s memory request is far below the limit, causing the scheduler to pack too many pods on a node. When the leak expands, the node reaches its allocatable threshold and the kubelet starts killing the biggest offender.

Fix: Align request/limit and enable resourceQuota to enforce caps at the namespace level.

apiVersion: v1
kind: ResourceQuota
metadata:
  name: memory-quota
spec:
  hard:
    requests.memory: "10Gi"
    limits.memory: "12Gi"

Symptom: “Readiness probe failed: Get http://…/healthz: dial tcp …: i/o timeout”

Why: The liveness probe runs before the model is fully loaded, causing a false negative that restarts the pod repeatedly.

Fix: Add a startup probe that waits for the model file to appear, and make the readiness probe check a lightweight endpoint (e.g., /ready that returns true once the model is in memory).

startupProbe:
  httpGet:
    path: /startup
    port: 8080
  failureThreshold: 30
  periodSeconds: 5

Symptom: “Failed to schedule pod: insufficient memory”

Why: HPA scales based on CPU while the real bottleneck is memory, so the scheduler tries to place more pods than the node pool can hold.

Fix: Switch HPA metric to resource.memory and use a node selector for nodes with larger allocatable.memory.

nodeSelector:
  gpu: "true"   # if you have GPU nodes with more RAM

Symptom: “Too many open files”

Why: Unclosed HTTP responses leave file descriptors dangling. The default http.Client reuses connections, but if you never read resp.Body to EOF, the FD stays.

Fix: Always defer resp.Body.Close() and read the body fully, even on error paths.

resp, err := client.Do(req)
if err != nil {
    return nil, err
}
defer resp.Body.Close()
body, err := io.ReadAll(resp.Body)
if err != nil {
    return nil, err
}

Frequently asked questions

If you’ve faced a memory‑leak panic in your own LLM service, drop a comment with the pattern that saved you. I’ll add it to the next post. Happy debugging!