AI Agent Circuit Breakers for Harness: Stop Outages (2026)

I was on call at 02:17 am, staring at a bright‑green “deployment succeeded” banner in Harness, while downstream services were spitting 500s. The culprit? A GenAI agent that had just generated a Terraform module with a missing `provider` block. The pipeline kept rolling, the infra team got paged, and we spent two hours rolling back manually before the breaker finally tripped. The whole episode taught me two things: **AI‑driven steps are not magical, and you need a hard stop before they blow up the whole system.**

AI agent circuit breakers in Harness workflows are resilience patterns that prevent automated AI deployment steps from causing cascading failures. They monitor for timeouts, errors, or malformed AI output, trip to halt requests, and enable a fallback path, ensuring graceful degradation. Implementation involves configuring state management, error thresholds, and human‑in‑the‑loop approvals.

The Critical Role of Resilience in AI‑Driven Deployments

The rise of GenAI agents in CI/CD

Since 2023 the hype around “self‑healing pipelines” has turned into production reality. Harness now ships an **AI Agent Plugin** that can call OpenAI or Claude, synthesize IaC, and push it downstream—all without a human typing a line. In our org we use the plugin to generate Helm values for each feature branch, auto‑scale Terraform workspaces, and even draft PR descriptions. The speed gain is undeniable, but the failure surface has changed.

Why blind automation creates ticking time bombs

Traditional scripts fail hard when a command returns a non‑zero exit code. AI agents, however, can return *syntactically correct* JSON that encodes a wrong security group, or they can silently drop a required field. Those “semantic” failures don’t trigger a non‑zero exit, so Harness treats the step as a success and progresses downstream. The next stage might try to apply the config, hit a quota error, and cascade into a full‑service outage. In short, **the error is invisible until it hurts**.

From chaotic failures to graceful degradation

Circuit breakers were invented for micro‑service communication, but the same principle applies when the *client* is an AI. By wrapping the agent call in a state machine you can:

• Stop sending requests once a failure rate crosses a threshold.
• Give the downstream services a chance to recover (cool‑down).
• Flip to a safe fallback – e.g., a cached IaC template or a manual approval.

That shift from “let’s keep going” to “let’s pause and ask a human” is the difference between a noisy incident and a quiet, reversible event.

—

Architectural Blueprint for a GenAI Circuit Breaker in Harness

Core states: Closed, Open, Half‑Open

The classic three‑state model still works, but we augment it with **AI‑specific signals**:

State	When we’re here	What we do
Closed	Failure rate < 3 % over last 5 min, output validation passes	Forward every request to the agent.
Open	Failure rate ≥ 3 % **or** semantic validation fails 3 times in a row	Reject all new requests, emit a “breaker tripped” event.
Half‑Open	Cool‑down elapsed (default 2 min)	Send a *probe* request; if it succeeds, move to Closed; otherwise back to Open.

The thresholds are not static. In practice we expose them as pipeline variables (`breaker_error_rate`, `breaker_cooldown_secs`) so each service can tune its own resilience.

Integrating with Harness’ Approval and Execution Framework

Harness 2026 introduced **Approval Steps API v4**, which lets you programmatically create, resolve, or reject approvals from within a workflow. We can harness (pun intended) that to turn an Open state into a **human‑in‑the‑loop reset**:

# harness.yaml – snippet
- step:
    name: "GenAI Deployment"
    type: "CUSTOM"
    spec:
      image: "myorg/ai-agent:latest"
      commands:
        - ./run-agent.sh
    failureStrategies:
      - onFailure:
          actions:
            - type: "CALLBACK"
              spec:
                url: "https://ci.myorg.com/breaker/notify"

When the circuit state flips to Open, the callback invokes a tiny service that creates a Harness Approval step via the v4 API. The on‑call engineer can inspect the generated IaC, click **Approve** (which resets the breaker) or **Reject** (which keeps the circuit Open). This pattern gives us an audit trail and a safety valve without needing a separate ticketing system.

AI‑specific state transition triggers (beyond simple error counts)

Pure error counts work for network timeouts, but AI agents need extra guards:

• **Semantic validation failures** – we run a lightweight JSON schema check on the output. Three consecutive schema violations push the breaker Open.
• **Cost‑budget breaches** – if the cumulative token usage for the last minute exceeds a budget, we treat it as a throttling event and open the circuit.
• **Model degradation alerts** – the provider can emit a `degraded` flag (Claude does this). That flag is a hard trigger to Open.

These triggers are implemented as **hooks** in the Harness delegate, written in Java using Resilience4j’s `CircuitBreaker` decorator. The delegate inspects the response payload before deciding whether to mark the call as success or failure.

—

Step‑by‑Step Implementation in YAML (Harness 2026 Best Practice)

Defining the circuit breaker policy

First, declare a **global policy** that the delegate will read. The policy lives in a Harness secret (so you can change thresholds without redeploying pipelines).

# policies/ai-breaker-policy.yaml
# version: Harness CD Community Edition 2026.03
breaker:
  errorRateThreshold: 0.03       # 3 % failures
  minimumRequests: 20
  coolDownSeconds: 120
  semanticFailureThreshold: 3   # consecutive schema violations
  costBudgetTokensPerMin: 5000

Load this secret in the pipeline:

- step:
    name: "Load Breaker Config"
    type: "SET_VARIABLE"
    spec:
      variables:
        - name: "breaker_cfg"
          valueFrom:
            secret: "ai-breaker-policy"

Wrapping your AI agent step

Next, wrap the actual AI call with a **circuit‑breaker interceptor**. The interceptor is a small Docker image that uses Resilience4j to enforce the policy.

# harness.yaml – full pipeline excerpt
- step:
    name: "AI Agent with Breaker"
    type: "CUSTOM"
    spec:
      image: "myorg/ai-breaker-interceptor:2.4"   # built on Go 1.24, Resilience4j 2.1
      env:
        - name: "POLICY_JSON"
          value: "${breaker_cfg}"
        - name: "OPENAI_API_KEY"
          valueFrom:
            secret: "openai-key"
      commands:
        - |
          #!/usr/bin/env bash
          set -euo pipefail
          # The interceptor expects the target command as args.
          exec /interceptor/run.sh ./run-ai-agent.sh
    onFailure:
      actions:
        - type: "CALLBACK"
          spec:
            url: "https://ci.myorg.com/breaker/open"

`run-ai-agent.sh` is the script that talks to OpenAI/GPT or Claude, writes the generated IaC to `output.yaml`, and exits with `0` if the provider returns HTTP 200 **and** the JSON schema passes. The interceptor maps a schema violation to a `Resilience4jCircuitBreakerOpenException`, which Harness records as a failure and triggers the on‑failure callback.

Implementing the fallback workflow

When the circuit opens we need a **fallback path** that either uses a cached manifest or hands off to a manual step.

- step:
    name: "Fallback IaC Provider"
    type: "CUSTOM"
    when:
      condition: "${pipeline.variables.breaker_state == 'OPEN'}"
    spec:
      image: "myorg/cached-iac-provider:1.0"
      commands:
        - cp /cache/last_good.yaml output.yaml
- step:
    name: "Manual Review (Human‑in‑the‑Loop)"
    type: "APPROVAL"
    when:
      condition: "${pipeline.variables.breaker_state == 'OPEN'}"
    spec:
      approvers:
        - "team-leads"
      timeout: "30m"
      description: "Breaker opened. Review generated IaC before proceeding."

The `when` clause reads a variable the delegate populates (`breaker_state`). If the circuit is Open, Harness skips the AI step, runs the cached provider, and then blocks on a manual approval. Once the approver clicks **Approve**, a small webhook fires back to the delegate, resetting the internal circuit state and allowing the next pipeline run to go **Closed** again.

**My take:** It feels tempting to “just add retries”. In my experience retries on a flapping LLM are a nightmare – you waste token budget, you amplify latency, and you still won’t catch a malformed schema. The breaker + manual approval combo is the only pattern that kept my team from a week‑long outage in Q2 2026.

—

Production‑Grade Error Handling and Monitoring

Logging for AI decision‑making audit trails

Every request that passes through the interceptor writes a JSON line to the Harness delegate log:

{
  "timestamp":"2026-08-28T14:03:21Z",
  "requestId":"c7f5e1ab-9d23-4f2a-8b1e-7a9f2d3c6b8a",
  "breakerState":"CLOSED",
  "provider":"openai-gpt-4",
  "tokensUsed":124,
  "validation":"PASSED",
  "latencyMs":87
}

These logs ship to **Prometheus** via a side‑car exporter and to **Harness SRM** as custom metrics (`ai_breaker_state`, `ai_validation_failure_total`). The detailed audit trail lets you reconstruct **who** approved a fallback and **why** the breaker opened.

Alerting on breaker trips vs. resource exhaustion

Set up two alert families:

Alert	Condition	Action
`BreakerTrip`	`ai_breaker_state == 1` (Open) for > 30 s	PagerDuty page “AI Ops” rotation
`HighTokenSpend`	`sum(rate(ai_tokens_used[1m])) > 5000`	Slack webhook to cost‑ops channel

Both alerts live in **Harness SRM 24.10**, with a secondary alert on `breaker_state_flapping_total > 5` to catch badly tuned thresholds. Flapping is a classic sign that the error window is too small.

The Human‑in‑the‑Loop escalation path

When the breaker opens, the delegate fires a POST to `https://ci.myorg.com/breaker/open`. That endpoint does three things:

1. Creates a Harness Approval step via **Approval Steps API v4**.
2. Posts a message to the **#ai-ops** Slack channel with a link to the pipeline run.
3. Stores the request ID in a Redis cache for later correlation.

If the approver resets the circuit, the delegate receives a webhook `breaker/reset` and flips the internal state to **Closed**. The whole loop completes in under 5 seconds, far quicker than a manual incident response that would otherwise involve digging through Terraform plan diffs.

—

Architectural Trade‑Offs and Performance Benchmarks

Latency overhead: When is the breaker too expensive?

We ran a benchmark on a 30‑node Kubernetes cluster (k8s 1.31, Harness Delegate 23.07). The baseline AI call (raw GPT 4 request) averaged **92 ms p95**. Adding the Resilience4j interceptor added **+18 ms** (19 % overhead). The fallback path added **+7 ms** when the circuit was Closed, but when Open the latency dropped to **~30 ms** because the interceptor short‑circuits the request.

Scenario	p95 latency (ms)
Raw AI call	92
With circuit breaker (Closed)	110
With breaker (Open, cached)	30
With fallback + approval wait	30 + human delay

We concluded that a **< 150 ms** p95 impact is acceptable for most CI/CD workloads, provided you tune the `coolDownSeconds` to avoid unnecessary Open periods.

Resource consumption vs. blast radius

Running the interceptor on each delegate consumes ~20 mCPU and 30 MiB RAM per pipeline. That’s tiny compared to a full‑blown LLM pod, but when you have 200 concurrent pipelines the aggregate can become noticeable. The trade‑off is clear: a few extra millicores prevent a **full‑cluster rollback** that would cost hours of engineer time.

Case analysis: Coordinating multiple breakers in complex DAGs

In a monorepo we have three independent AI steps: *feature‑branch env*, *security‑group generation*, and *cost‑estimate*. Each step owns its own breaker, but they share a **global cooldown bucket**