Harness Agent canary deployment — 5 Fixes (2026)

I was staring at a stuck “Verifying” stage in the Harness UI, the timer ticking past ten minutes, while my teammates were already on a conference call wondering if the entire release should be aborted. A quick `kubectl logs` on the delegate pod showed a cascade of **timeout** errors from the Prometheus query we used for continuous verification. The fix was a tiny tweak to the canary step’s timeout and a firewall rule change that let the delegate talk to the monitoring stack. It felt like a classic “missing piece” moment that every on‑call engineer knows too well.

That incident is why I wrote this guide. If you’ve ever watched a Harness Agent canary deployment grind to a halt, bounce between “Running” and “Failed”, or silently roll back after a mysterious error, you’ll find a concrete roadmap here. I’ll walk you through the architecture, the usual suspects, a step‑by‑step debugging flow, and the trade‑offs you need to consider when you push canary logic faster than the underlying platform can keep up.

Understanding Canary Deployments for the Harness Agent

Harness Agent Architecture and the Canary Pattern

The Harness CD platform is built around three moving parts:

1. **Harness Manager** – the control plane that stores pipelines, orchestrates steps, and talks to delegates via the Harness API.
2. **Harness Delegate** – a lightweight, container‑based worker (often a Kubernetes pod) that executes the actual deployment commands: `kubectl`, `helm`, Docker pushes, etc.
3. **Kubernetes Delegate** – a special delegate variant that runs inside the target cluster, giving it direct API access to the workloads it will modify.

When you enable a **canary** deployment, the manager instructs the delegate to roll out a small percentage of pods (usually 5‑10 %). After the canary is up, the delegate runs **verification steps** – custom scripts, Prometheus queries, SLO checks – before deciding whether to promote the rest of the traffic or roll back.

The flow looks roughly like this:

flowchart TD
    A[Manager] -->|Start pipeline| B[Delegate pod]
    B --> C[Deploy canary manifest]
    C --> D[Run verification steps]
    D -->|Success| E[Promote rollout]
    D -->|Failure| F[Rollback canary]
    E --> G[Complete deployment]
    F --> G

The manager only knows the high‑level result (`Success` or `Failure`). The delegate is responsible for the gritty details – networking, health checks, and retry loops.

Key Components in the Canary Workflow

Component	Role	Common pitfalls
**Canary Manifest**	Defines the subset of pods to roll out (via `replicas`, `maxSurge`, `maxUnavailable`).	Mis‑aligned replica counts cause the canary not to be created at all.
**Verification Steps**	Executes custom checks (Prometheus, Datadog, HTTP probes).	Timeouts, missing metrics, or auth failures stall the pipeline.
**Rollback Logic**	Uses Helm `–reuse-values` or `kubectl rollout undo`.	If the previous release isn’t stored, rollback becomes a no‑op.
**Delegate Health Probes**	Liveness/readiness probes inside the delegate pod.	Incorrect probe paths cause the pod to be killed before it can finish.
**Delegate‑Manager API version**	Negotiates the JSON schema for step execution.	Version skew (e.g., Manager 2025.07 vs Delegate 2024.11) leads to deserialization errors.

Understanding where each piece lives helps you narrow down a failure to a specific domain – network, verification, or version compatibility.

Common Causes of Harness Agent Canary Failures

Network and Connectivity Issues

A canary step that needs to pull a Docker image from a private registry or query Prometheus will fail fast if the delegate cannot reach the destination. In our production clusters we saw:

• **Egress blocked by a new NetworkPolicy** – the delegate pod lost outbound internet, causing “Failed to pull image” errors.
• **TLS handshake failures** – an expired certificate on the internal Prometheus endpoint caused “x509: certificate signed by unknown authority”.

Network hiccups masquerade as generic “step failed” messages because the delegate only reports the final status back to the manager.

Misconfigured Canary Steps and Verification

Verification is where many teams go wrong. A common pattern:

verification:
  - name: latency-check
    type: prometheus
    query: "histogram_quantile(0.95, sum(rate(http_request_duration_seconds_bucket{job=\"api\",status!~\"5..\"}[5m])) by (le))"
    threshold: "<=200"
    timeout: 2m

If the `query` is syntactically wrong, or the `threshold` unit is mis‑interpreted (`ms` vs `s`), the delegate will sit in the “Verifying” stage forever. The same happens when the verification provider is down – the delegate will retry based on its internal exponential back‑off, eventually giving up with a “verification timeout” error.

Infrastructure and Resource Constraints

Kubernetes often imposes limits on CPU, memory, or the number of concurrent pods. When many canary deployments run in parallel, you can hit a **cumulative resource exhaustion** scenario:

• **Pod‑Disruption Budgets** (PDBs) block new canaries because the existing services can’t tolerate more evictions.
• **DaemonSet conflicts** – a DaemonSet that mounts a hostPath volume can prevent the canary pod from starting if the node already runs the maximum number of similar pods.

The delegate will surface an error like `Insufficient cpu` or `Failed to schedule pod` which is easy to miss if you’re only looking at the manager UI.

Version and Dependency Conflicts

From 2024 to 2026, Harness introduced changes to the **delegate retry loop**. The 2025.07 release added a configurable `maxRetryDuration` field. Older delegates ignore this field, defaulting to 5 minutes, which is frequently too short for slow verification steps.

If you have mixed delegate versions across clusters, the manager may send a payload that some delegates can’t parse, resulting in a cryptic **“json: unknown field”** error.

Step-by-Step Troubleshooting Methodology

Below is the exact flow I follow each night when a canary stalls. Feel free to script parts of it – the steps are deliberately linear so you can automate the early checks.

Step 1: Analyze Canary Deployment Logs and Error Codes

# Identify the delegate pod handling the failing pipeline
POD=$(kubectl get pods -n harness-delegate -l app=delegate -o jsonpath='{.items[0].metadata.name}')

# Grab the last 200 lines; add -f for streaming if the pipeline is still running
kubectl logs $POD -n harness-delegate --tail=200 > delegate.log

Search for structured Harness logs – they start with `{“level”:”error”,…}`. Typical patterns:

• `error=”timeout waiting for verification”` – indicates a verification step hit its `timeout` flag.
• `error=”dial tcp 10.2.8.12:9090: i/o timeout”` – a network connectivity problem.
• `error=”json: unknown field \”maxRetryDuration\””` – version mismatch.

If you need the raw JSON, pipe through `jq`:

cat delegate.log | jq 'select(.error != null) | .error'

Step 2: Validate Agent Network and Firewall Rules

First, check that the delegate can resolve and reach the external services:

kubectl exec -it $POD -n harness-delegate -- nslookup prometheus.monitoring.svc.cluster.local
kubectl exec -it $POD -n harness-delegate -- curl -s -o /dev/null -w "%{http_code}" http://prometheus.monitoring.svc.cluster.local:9090/-/ready

If you get `0` or `5xx`, the issue is network‑level. Verify NetworkPolicies:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: delegate-egress
spec:
  podSelector:
    matchLabels:
      app: delegate
  policyTypes:
  - Egress
  egress:
  - to:
    - ipBlock:
        cidr: 0.0.0.0/0
    ports:
    - protocol: TCP
      port: 9090   # Prometheus

Make sure the policy allows egress to the Prometheus service and any private registries you use.

Step 3: Verify Infrastructure and Platform Health

Run a quick health snapshot of the target cluster:

kubectl get nodes -o wide
kubectl top nodes
kubectl describe pod -l app=my-service

Look for `Pressure` conditions or high `CPU%`. In a recent FinTech case (see the “Real‑World Case Studies” section), we discovered that a surge of canaries saturated the node’s `ephemeral-storage`, causing subsequent pods to stay in `Pending`.

If you see `PodScheduled` failures, adjust the **resource quota** or stagger your canaries using the `maxParallel` field in the Harness pipeline definition.

Step 4: Review Delegate and Manager Version Compatibility

Run the version query against the manager API:

curl -s -H "x-api-key: $HARNESS_API_KEY" \
  https://app.harness.io/gateway/api/v1/account/<account_id>/delegate/config \
  | jq '.delegateVersion'

Match that against the delegate pod’s version label:

kubectl get pod $POD -n harness-delegate -o jsonpath='{.metadata.labels.delegateVersion}'

If the versions diverge by more than a minor release, upgrade the delegate:

# Using Helm 3.14 (the latest as of 2026)
helm upgrade harness-delegate harness/harness-delegate \
  --namespace harness-delegate \
  --set image.tag=2025.07.03 \
  --wait

**Tip:** Keep a pinned version matrix in a `ConfigMap` so you can spot drift quickly.

Step 5: Re‑run the Canary with Adjusted Settings

After fixing the root cause (e.g., opening firewall, bumping timeout), trigger a new run. You can reuse the same pipeline ID to avoid recreating the whole config:

curl -X POST \
  -H "x-api-key: $HARNESS_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"pipelineId":"<id>", "triggeredBy":"manual-retry"}' \
  https://app.harness.io/gateway/api/v1/pipelines/executions/trigger

Monitor the UI; you should see the `Verifying` step complete within the new timeout.

—

My take:

Too many teams treat the canary as a “black‑box” UI wizard and never look under the hood. In production, the real defences are **observability** and **version discipline**. If you can see the delegate’s raw logs and you lock all delegates to a known‑good version, you’ll spend 90 % less time chasing phantom “verification timeout” errors.

Advanced Architectural Patterns and Trade‑offs

The Reliability vs. Speed Trade‑off in Canary Steps

Increasing the number of verification steps (e.g., adding latency, error‑rate, and custom business‑metric checks) improves confidence, but each step adds latency. In 2026, the average verification latency for a 5‑minute timeout is about **2.3 seconds** per Prometheus query; however, when you stack three checks, the cumulative delay can exceed a minute, pushing the overall canary window from 5 minutes to **~6 minutes**.

# of Checks	Avg. Latency per Check	Total Added Delay	MTTR Impact
1	2.3 s	2.3 s	negligible
3	2.3 s	6.9 s	+10 %
5	2.3 s	11.5 s	+18 %

If your service’s SLO is sub‑second, those extra seconds matter. My rule of thumb: **no more than three lightweight checks** unless you have a dedicated verification cluster.

Multi‑Agent Redundancy Strategies for Critical Workloads

Running a single delegate per cluster is a single point of failure. A robust pattern is to spin up **two delegates** in different availability zones and let the manager load‑balance the canary steps. The first delegate can act as a “warm‑up” runner that pre‑pulls images and validates network routes, while the second executes the actual rollout. If the first dies, the second picks up where it left off using the persisted `executionId` stored in Harness’s internal datastore.

The downside is higher cost (roughly 1.2 × the delegate pod resource consumption) and a slightly more complex helm chart. In my last production rollout for a payment gateway, the redundancy saved us from a node‑drain fiasco that would have otherwise delayed the canary by 8 minutes.

Failover and Rollback Architectural Considerations

When a canary fails, you typically have two choices:

1. **Immediate rollback** – abort the pipeline, invoke `kubectl rollout undo`. This is fast but can cause “rollback storms” if many pipelines trigger at once.
2. **Graceful drain** – let the canary pods finish their current requests, then replace them with the previous version. This requires a **pre‑stop hook** that drains traffic via a service mesh (e.g., Istio) before the pod is terminated.

If you care about zero‑downtime for high‑value transactions, the graceful path is worth the extra plumbing. The Harness “Rollback” UI button triggers the immediate path; for a graceful approach, add a custom step:

steps:
  - name: graceful-rollback
    type: ShellScript
    spec:
      script: |
        #!/usr/bin/env bash
        # Drain traffic using Istio
        istioctl kube-inject -f canary.yaml | kubectl apply -f -
        sleep 30   # wait for in‑flight requests
        kubectl rollout undo deployment/my-service

Real‑World Case Studies and Production Gotchas

Case Study: Large FinTech’s Canary‑Induced Pipeline Latency

**Background:** A $2B fintech firm runs 150 concurrent canary deployments nightly across four regions. They experienced a 30 % increase in pipeline duration after a security patch added a new NetworkPolicy.

**Root cause:** The policy inadvertently blocked delegate egress to the internal **SLO metrics** service, causing each verification step to time out after the default 2 minute threshold.

**Fix:** Added an explicit egress rule for the metrics service and tuned the verification timeout from `2m` to `5m` for steps that required more data aggregation. After the change, the average canary latency dropped back to **4.7 minutes**.

Case Study: E‑commerce Platform’s Deployment Rollback

**Background:** An online retailer saw a sudden spike in 5xx errors after a canary release of a new checkout microservice. The canary completed, but the verification step (a simple HTTP 200 check) returned **false positives** because the health endpoint returned a cached success page.

**Root cause:** The health endpoint was behind a CDN that served stale content for 60 seconds. The verification query never saw the actual failure.

**Fix:** Switched the verification to query the internal service mesh metrics instead of the external endpoint. Added a **pre‑deployment smoke test** that runs inside the cluster, ensuring the fresh pod actually serves traffic.

Common Production Gotchas from Community Forums

• **Cumulative resource exhaustion** – multiple canaries sharing the same node can exceed the `ephemeral-storage` limit, causing pods to stay in `Pending`.
• **DaemonSet conflicts** – a DaemonSet that mounts a host