Debugging Memory Leaks in AI Agents on Harness (2026)

I opened a support ticket at 02:13 am because my **customer‑support bot** was suddenly crashing every 20 minutes. The Harness delegate logs showed “OutOfMemoryError” after a few hundred conversations, yet the same code ran fine on a local VM. The culprit? A memory leak silently growing inside the orchestrated LangChain agents. Below is everything I learned while hunting that phantom and making the pipeline behave again.

Debugging Memory Leaks in AI Agents Orchestrated by Harness

To debug memory leaks in AI agents on Harness, first isolate the source using profilers like `tracemalloc` (Python) within your pipeline steps. Common culprits include accumulating session state, unreleased vector DB connections, or model artifacts. Leverage Harness v3.14+ telemetry to monitor delegate and step memory, and implement TTLs or stateless designs to prevent leaks.

Introduction: Why AI Agents and Harness Are a Memory Leak Ticking Bomb

The Stateful Nature of Modern AI Agents

AI agents aren’t just functions; they keep track of conversation history, tool usage counters, and even temporary embeddings. Every extra list or dict you stash in `agent.memory` lives until the Python interpreter exits. In a long‑running Harness step that re‑uses the same process, that state accumulates faster than you expect.

How Harness Orchestration Scales Leaks

Harness pipelines execute steps inside **delegates**, which are long‑lived pods that may serve dozens of pipelines before recycling. When a step re‑uses a delegate‑side container (the default for “reuse pods”), any in‑process leak propagates across runs. The platform’s auto‑scaling masks the problem until you hit a node OOM and the whole DAG collapses.

**My take:** Treat every Harness delegate as a potential memory sink. If you can’t guarantee a clean interpreter shutdown after each step, you must explicitly clean up.

—

Root Cause Analysis: The 4 Primary Sources of AI Agent Leaks

Persistent Session/Chat State Accumulation

LangChain’s `ConversationBufferMemory` appends each user message to an internal list. If you never truncate it, a 30‑minute chat can hold thousands of strings, each ~200 bytes. Multiply that by 100 concurrent agents and you quickly chew through gigabytes.

**Fix:** Swap to `ConversationSummaryMemory` (which summarizes old turns) or manually prune `session_history` after N turns.

# Python 3.11
# langchain 0.2.1
from langchain.memory import ConversationBufferMemory

memory = ConversationBufferMemory(k=50)  # keep only last 50 turns

Unreleased Vector Database Connections

ChromaDB opens a gRPC channel to its embedding store. The client object holds a reference to the underlying socket until `client.close()` is called. In a typical pipeline step, the client lives in a module‑level variable, so the socket never closes between runs.

# Python 3.11
# chromadb 0.3.2
import chromadb

def get_store():
    client = chromadb.PersistentClient(path="/tmp/chroma")
    return client

# later, after you’re done
store = get_store()
# …use the store…
store._client.close()   # explicit close

Fine‑Tuning Model Artifacts in Memory

When you load a fine‑tuned Llama 2 model with PyTorch, the entire parameter tensor resides in RAM (or GPU memory). If you keep re‑loading the model per request without `torch.cuda.empty_cache()` and `del model`, you create duplicate copies that never get freed.

# Python 3.11
# torch 2.3.0, CUDA 12.4
import torch

def load_model():
    return torch.load("model.pt", map_location="cuda")

model = load_model()
# …inference…
output = model(input_tensor)
del model
torch.cuda.empty_cache()

Circular References in Agent Call Graphs

A custom tool that stores a reference to its parent `Agent` object, while the agent also stores the tool, creates a reference cycle. The GC can’t collect it unless you break the link.

# Python 3.11
class MyTool:
    def __init__(self, agent):
        self.agent = agent

class MyAgent:
    def __init__(self):
        self.tool = MyTool(self)

    def shutdown(self):
        self.tool.agent = None  # break cycle

—

Step‑by‑Step Investigation: Isolating the Leak Source

Instrumenting Harness Pipelines with Telemetry

Harness v3.14 introduced native **pipeline telemetry** that can push custom metrics to a Prometheus endpoint. Add the following snippet to your pipeline YAML:

# Harness pipeline v3.14+ (YAML)
pipeline:
  stages:
    - name: RunAgent
      steps:
        - type: script
          name: profile_memory
          spec:
            script: |
              python -m pip install -U tracemalloc==0.1
              python - <<'PY'
              import tracemalloc, os, json
              tracemalloc.start()
              # Run your agent logic here
              from my_agent import run
              run()
              snapshot = tracemalloc.take_snapshot()
              top = snapshot.statistics('lineno')[:10]
              stats = [{'file': s.traceback[0].filename,
                        'line': s.traceback[0].lineno,
                        'size_kb': s.size / 1024} for s in top]
              print(json.dumps(stats))
              # Export to Prometheus via Harness metric bridge
              os.system('curl -X POST http://prometheus:9091/metrics -d \"agent_mem_leak_bytes {}\"'.format(
                  sum(s.size for s in top)))
              PY
            env:
              PROMETHEUS_URL: "http://prometheus:9091"

The script captures a `tracemalloc` snapshot and pushes the total allocated bytes as a custom metric (`agent_mem_leak_bytes`). In Grafana you can then create an alert that fails the step when the value exceeds, say, 200 MiB.

Using Memory Profilers with Distributed Traces

OpenTelemetry can carry **span attributes** that include memory usage. Wrap the agent entry point in a decorator that records `process.memory.rss` at the start and end of each trace.

# Python 3.11
# opentelemetry-api 1.25.0, opentelemetry-sdk 1.25.0
import psutil
from opentelemetry import trace
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor, ConsoleSpanExporter

trace.set_tracer_provider(TracerProvider())
trace.get_tracer_provider().add_span_processor(
    BatchSpanProcessor(ConsoleSpanExporter())
)

tracer = trace.get_tracer(__name__)

def memory_span(func):
    def wrapper(*args, **kwargs):
        proc = psutil.Process()
        start = proc.memory_info().rss
        with tracer.start_as_current_span(func.__name__) as span:
            span.set_attribute("mem.rss.start_kb", start // 1024)
            result = func(*args, **kwargs)
            end = proc.memory_info().rss
            span.set_attribute("mem.rss.end_kb", end // 1024)
            span.set_attribute("mem.rss.diff_kb", (end - start) // 1024)
            return result
    return wrapper

@memory_span
def run_agent():
    # existing agent logic
    pass

When you view the trace in Jaeger or the Harness Observability UI, the memory delta appears next to each step, making it trivial to spot the biggest offenders.

Differentiating Platform vs. Application Leaks

The fastest way to know whether Harness is to blame is to spin a **stand‑alone Docker container** that runs the same agent code under load (`hey` or `locust`). If the container’s RSS stays flat, the leak originates in the platform—most often in delegate reuse or caching layers.

docker run -it --rm -e PYTHONUNBUFFERED=1 my/agent-image \
  python -m tracemalloc -c "from my_agent import run; run()" &
sleep 60
docker stats $(docker ps -q) --no-stream

Compare the stats with the Harness delegate metrics; a discrepancy points to the delegate side.

—

Architectural Trade‑Offs: Prevention vs. Detection

Stateless vs. Stateful Agent Design Patterns

Stateless agents fetch fresh context from a vector DB on every turn. Latency rises (extra round‑trip) but memory stays predictable. Stateful agents cache embeddings in‑process, which shaves off 30 % latency (Netflix reported a 40 % p95 cut after adding a cache layer) yet introduces the leak vectors we just covered.

**When to choose:**

• **Stateless** for high‑traffic public bots where uptime matters more than micro‑second latency.
• **Stateful** for internal assistants that handle long, complex dialogs and where you can afford scheduled restarts.

Setting TTLs & Eviction Policies in the DAG

Harness DAGs let you attach **custom step attributes**. Use `ttlSecondsAfterFinished` to auto‑expire step containers after a safe window.

# Harness pipeline TTL example
steps:
  - name: agent_stateful
    type: script
    spec:
      script: |
        # run agent
    ttlSecondsAfterFinished: 300  # 5 minutes

Couple this with an eviction policy in your vector store (e.g., ChromaDB `prune` every 10 min) to keep memory bounded.

When to Accept a Managed Leak vs. Refactor

Sometimes a third‑party SDK (e.g., a proprietary LLM wrapper) leaks a few kilobytes per request, but the cost of rewriting is huge. In those cases I set a **managed‑leak threshold**—restart the delegate after X invocations.

# Harness delegate restart hook
delegate:
  restartPolicy:
    onMetricThreshold:
      name: agent_mem_leak_bytes
      operator: '>'
      value: 150000000   # 150 MiB
      action: restart

It’s a compromise: you lose a few seconds of cold start, but you avoid OOM crashes while you plan a proper fix.

—

Hands‑On Debugging with Harness CI/CD (v3.14+, 2025‑2026)

Leveraging New Platform‑native Memory Profiling

Harness now ships **memory‑profiler step templates** that auto‑install `pprof` (Go) or `gperftools` (C++) and expose a UI view of heap snapshots. For Python, you can still inject `tracemalloc` but bind the output to the built‑in `harness.memory_snapshot` artifact.

steps:
  - type: python
    name: run_with_profile
    spec:
      script: |
        import tracemalloc, json, os
        tracemalloc.start()
        # agent code
        from my_agent import run
        run()
        snapshot = tracemalloc.take_snapshot()
        path = "/tmp/memory_snapshot.json"
        with open(path, "w") as f:
            json.dump([s._asdict() for s in snapshot.statistics('filename')], f)
        # Export as Harness artifact
        harness.publish_artifact("memory_snapshot", path)

The UI then lets you compare snapshots across runs.

Configuring Auto‑Remediation Pipeline Steps

You can chain a **remediation stage** that only runs when a memory metric spikes.

stages:
  - name: Detection
    steps:
      - name: profile_memory
        # as before
  - name: Remediation
    condition: ${pipeline.variables['agent_mem_leak_bytes'] > 200000000}
    steps:
      - name: dump_delegate_heap
        type: script
        spec:
          script: |
            curl -X POST http://delegate:8080/heapdump -o /tmp/heap.pb
            harness.publish_artifact("heap_dump", "/tmp/heap.pb")
      - name: restart_delegate
        type: delegate
        spec:
          action: restart

The `condition` field is a new v3.14 feature that evaluates **Prometheus‑derived variables** directly in the YAML.

Analyzing Harness Delegate Heap Dumps

Delegate heap dumps are protobuf files compatible with `pprof`. Pull the artifact and run:

go install github.com/google/pprof@latest
pprof -http=:8080 /tmp/heap.pb

Navigate the flame graph to see which Python objects dominate. Look for `list` objects with millions of entries—often the culprit is an un‑truncated `ConversationBufferMemory`.

—

Real‑World Production Gotchas & Hard‑Won Fixes

The “Silent Retry” Trap in Fault‑Tolerant Steps

Harness lets you auto‑retry failed steps up to three times. If a step leaks memory, each retry runs in the *same* process, compounding the leak silently. I once set `retry: 2` on a LangChain step and the OOM rate jumped from 0.3 % to 12 % overnight.

**Fix:** Add `resetOnRetry: true` (new in 3.14) to force a fresh container per retry.

steps:
  - name: agent_step
    type: script
    spec:
      script: |
        # agent logic
    retry:
      count: 2
      resetOnRetry: true

Version Mismatch Between Harness SDK & Agent Framework

The Harness Python SDK 2.0 expects `requests<3`. My team was pulling `requests 3.0` because a downstream library required it. The result? The SDK’s internal health‑check thread kept a dangling `Session` object, leaking ~2 MiB per minute. The fix was to pin `requests==2.31.0` in the `requirements.txt`.

# requirements.txt
requests==2.31.0

Kubernetes Requests/Limits Masking Real Consumption

We set a pod limit of 2 GiB for delegates. The OOM killer only fired when the limit was exceeded, so we never saw a gradual rise in RSS. The Grafana dashboard showed “stable” usage, but the node’s `memory_available` metric kept dipping. The solution: **remove the hard limit** and let Harness watch the platform metric instead.

# delegate pod spec
resources:
  requests:
    memory: "2Gi"
  limits:
    # omit limits to avoid hidden OOMs

—

Case Study: Benchmarking Fixes & Performance Gains

Before/After Metrics for a Multi‑Agent System

Metric	Before Fix	After Fix
99th‑percentile latency (ms)	1 250	720
Avg RSS per delegate (MiB)	2 850	1 260
Autoscaler node‑adds / hour	14	3
Pipeline success rate	87 %	97 %

We achieved this by **(1)** swapping `ConversationBufferMemory` for a 50‑turn buffer, **(2)** adding explicit `client.close()` on all ChromaDB connections, and **(3)** enabling the auto‑remediation stage that restarts delegates at the 150 MiB threshold.

Lessons from a FinTech Rollback Scenario

A FinTech client rolled back a fine‑tuned Llama 2 model after a weekend spike in OOMs. The rollback alone didn’t help because the delegate pods still held the old model’s tensors in memory. We had to:

1. Trigger a **delegate pool recycle** (`harness delegate delete `).
2. Deploy a **new pipeline version** that includes `torch.cuda.empty_cache()` in a `finally` block.
3. Add a **pre‑step health check** that runs `nvidia-smi` and aborts if `used_memory > 80