Kubernetes Agent Not Responding: 5 Debug Tips (2026)

I was on call when a brand‑new microservice started spitting “agent not responding” in the kube‑let logs. Within ten minutes the pod was stuck in CrashLoopBackOff, our autoscaler was firing, and the on‑call pager lit up like a Christmas tree. The scary part? The same probes had been running fine for weeks on the exact same image. What went wrong? Turns out a tiny change to a NetworkPolicy and a sub‑second liveness probe combined to starve the pod of the health‑check traffic it needed. By the time I traced the iptables rules, the cluster was already under load.

Debugging “agent not responding” errors in Kubernetes health checks

Debugging “agent not responding” errors in Kubernetes requires systematic checks: validate pod resource limits, examine kubelet and container logs for OOM or throttling, ensure network policies allow probe traffic, and confirm node disk pressure isn’t stalling the container runtime. Adjusting probe timeouts and implementing proper retry logic often resolves false positives.

Understanding Health Check Failures and System Impact

Business Consequences of Unstable Pods

An unstable pod is more than an annoying log line. It can trigger cascade failures: autoscalers spin up new instances, load balancers keep routing traffic to flapping pods, and SLOs crumble. A 2024 Sysdig report showed 23 % of production container incidents start with a mis‑configured liveness probe, and the knock‑on effect can shave hours off your MTTR.

Fundamental Components: Probes, Kubelet, and kube‑proxy

Kubernetes health checks are split into three moving parts:

Component	Role	Typical Failure Modes
Readiness/Liveness Probe	Executes HTTP/TCP/gRPC checks from the node	Timeout, wrong path, network policy blocks
kubelet v1.29+	Calls the probe, updates pod status	Missed heartbeats, node pressure
kube‑proxy (iptables/ipvs)	Routes Service → Pod traffic, also handles health‑check packets when using externalTrafficPolicy=Local	Stale rules, IPVS sync lag

If any link in that chain breaks, the pod appears “unhealthy” even though the container might be perfectly fine.

Root Cause Analysis: Step‑by‑Step Diagnostic Checklist

Below is the checklist I run on every “agent not responding” alert. Treat it as a sprint‑backlog item; you can tick boxes while you sip coffee.

1. Log Inspection and Event Correlation

# Grab the last 100 lines from the kubelet on the node hosting the pod
journalctl -u kubelet -n 100 --since "5m ago" | grep -i "healthcheck\|probe"

# Pull pod events for quick correlation
kubectl describe pod <pod-name> -n <ns> | grep -i "Failed"

Look for patterns like Readiness probe failed: Get http://10.244.0.15:8080/healthz: dial tcp 10.244.0.15:8080: connect: timeout. If you see node pressure: memory or disk pressure in the kubelet logs, the node itself is throttling the container runtime.

2. Validating Network and Security Policies

NetworkPolicy misconfigurations are the silent killers of probes. Run:

kubectl get networkpolicy -n <ns> -o yaml | grep -C3 "podSelector: {app: my-service}"

If the policy only allows traffic from the frontend namespace, the kube‑let’s health‑check traffic (originating from the node IP) will be dropped. The fix is either to add a rule for the node CIDR or to set podNetworkPolicy: false on the probe (available on Cilium 1.15).

Tip: When using Linkerd, enable policy: true in the sidecar injector to automatically open ports for health checks.

3. Verifying Resource Configuration and Limits

Pods that scrape the CPU or memory limit can experience OOMKilled or throttling that stalls the probe handler. Check the pod spec:

resources:
  limits:
    cpu: "500m"
    memory: "256Mi"
  requests:
    cpu: "200m"
    memory: "128Mi"

If cpu limits are too tight for a Java app performing a warm‑up, the liveness probe may never get a chance to execute. Increase the limit or add a startup probe to give the JVM time to boot.

Advanced Resolution Patterns with Real‑World Code

Implementing Robust Retry Logic and Timeout Management

Instead of a bare curl in an HTTP probe, wrap it in a tiny sidecar that retries with exponential back‑off and emits a custom metric. Here’s a Go 1.24 example that you can drop into a sidecar container:

// main.go – health‑check sidecar (Go 1.24)
package main

import (
	"context"
	"net/http"
	"os"
	"time"

	"go.opentelemetry.io/otel"
	"go.opentelemetry.io/otel/trace"
)

func main() {
	tracer := otel.Tracer("probe-sidecar")
	for {
		ctx, span := tracer.Start(context.Background(), "probe")
		check(ctx)
		span.End()
		time.Sleep(2 * time.Second) // interval matches pod spec
	}
}

func check(ctx context.Context) {
	req, _ := http.NewRequestWithContext(ctx, http.MethodGet, "http://localhost:8080/healthz", nil)
	client := &http.Client{
		Timeout: 1 * time.Second,
	}
	backoff := 100 * time.Millisecond
	for i := 0; i < 5; i++ {
		resp, err := client.Do(req)
		if err == nil && resp.StatusCode == http.StatusOK {
			// Export success metric via OpenTelemetry Collector
			return
		}
		time.Sleep(backoff)
		backoff *= 2
	}
	// If we get here, mark pod as unhealthy via a file that the main container watches
	_ = os.WriteFile("/tmp/failed", []byte("1"), 0644)
}

Mount /tmp/failed into the main container and have the app expose GET /probe that returns failed if the file exists. This decouples the kubelet’s probe from the internal HTTP server’s transient hiccups and gives you observability via OTel traces.

Adjusting Probe Configuration for Production Load

Old tutorials often suggest initialDelaySeconds: 5 and periodSeconds: 10. In 2026 production workloads, especially stateful services or KEDA‑scaled jobs, those defaults are too aggressive.

Workload	Recommended initialDelaySeconds	periodSeconds	failureThreshold
Stateless Go HTTP	15	5	3
Java Spring (large heap)	120 (use startup probe)	10	6
KEDA‑scaled consumer	30	3	5
gRPC‑heavy service (Linkerd)	20	2	4

My take: Smaller periods look nice on dashboards but they multiply kubelet traffic across the cluster. On a 5 k node fleet, sub‑second probes can add ~200 MiB/s of extra traffic to the API server. Trade‑offs matter more than “faster detection”.

Troubleshooting CNI and Service Mesh Interference

When you run Cilium with Tetragon enabled, you can trace the exact packet path of a probe:

cilium monitor --type trace --filter 'event_type=trace' --selector 'dest_port==8080 && src_ip==<node-ip>'

If the trace never shows a packet reaching the pod, the issue is at the CNI level. Common culprits:

• IP masquerade off – the probe’s source IP is the node’s IP, which may be denied by a Policy object.
• Linkerd sidecar missing transparent-proxy – health checks bypass the proxy and get dropped.

Enable cilium debug capture or Linkerd’s proxy-status endpoint (/proxy-metrics) to verify the flow.

Production‑Proven Architecture and Configuration

Optimal Readiness/Liveness Probe Settings for 2025

The secret sauce is multi‑window evaluation: rather than a single binary pass/fail, evaluate health over several windows (1 min, 5 min, 15 min) and weight them. LinkedIn’s SLO‑based approach reduced alert noise by 70 %. Implement it with OpenTelemetry and a custom Prometheus rule:

# prom-rule.yaml – SLO‑based health evaluation
groups:
- name: health-slo
  rules:
  - alert: PodUnhealthySLO
    expr: |
      sum_over_time(kube_pod_container_status_ready{namespace="prod"}[5m]) / 5 < 0.9
    for: 2m
    labels:
      severity: warning
    annotations:
      summary: "Pod {{ $labels.pod }} failing SLO readiness"

Scaling and Load Balancing Considerations

A thundering‑herd of failing probes can hammer the API server. Mitigate with probe caching on the kubelet: set --probe-cache-max-size=5000 (kubelet flag introduced in v1.28). Also, use IPVS mode (--proxy-mode=ipvs) for kernel‑level load balancing; it handles 10× more concurrent connections with lower latency than iptables.

Immutable Infrastructure and Canary Deployment Patterns

Never modify probes in‑place on a running release. Instead, bake probe configuration into the immutable container image or Helm chart, and roll out a canary with a different probe profile. If the canary passes, promote; otherwise, rollback before the probe traffic spikes the rest of the cluster.

Case Studies: How Top Engineering Teams Prevent Downtime

Optimizing Cluster Performance Under Load

At a large e‑commerce platform, the ops team observed a spike in Readiness probe failed events during a Black Friday sale. They:

1. Added a Tetragon eBPF program that logged latency of each health‑check packet.
2. Correlated the data in Grafana 10.2 with latency spikes from the upstream payment gateway.
3. Tuned the readiness periodSeconds from 5 s to 15 s for the payment microservice, cutting the probe volume by 70 % and eliminating false failures.

The result: zero pod‑restarts during the traffic surge.

Automated Remediation in CI/CD Pipelines

Using a Kyverno policy, they enforced that any pod spec with a liveness probe must also define a failureThreshold ≥ 4 and periodSeconds ≥ 5. The policy automatically rejects PRs that violate the rule, stopping a mis‑configured probe from ever reaching production.

# kyverno.yaml – guardrail policy
apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: enforce-probe-guardrails
spec:
  validationFailureAction: enforce
  rules:
  - name: require-safe-probe
    match:
      resources:
        kinds: ["Pod"]
    validate:
      message: "Liveness probe must have periodSeconds >=5 and failureThreshold >=4"
      pattern:
        spec:
          containers:
          - =(livenessProbe):
              periodSeconds: "?* >=5"
              failureThreshold: "?* >=4"

Common Errors & Fixes

Error: Readiness probe failed: Get http://10.244.2.3:8080/ready: dial tcp 10.244.2.3:8080: connect: connection refused

Why: The container hasn’t started listening on the expected port yet, or a NetworkPolicy blocks node‑to‑pod traffic. Fix:

readinessProbe:
  httpGet:
    path: /ready
    port: 8080
  initialDelaySeconds: 30   # give the app time to bind
  periodSeconds: 10
  failureThreshold: 6

Add a NetworkPolicy rule:

- podSelector:
    matchLabels:
      app: my-service
  ingress:
  - from:
    - ipBlock:
        cidr: 10.0.0.0/8   # node CIDR
    ports:
    - protocol: TCP
      port: 8080

Error: livenessProbe failed: HTTP probe failed with statuscode: 500

Why: The liveness endpoint returns 500 during a GC pause or when a downstream DB is overloaded. Fix: Switch to a startup probe for the first 2 minutes, then a lighter liveness probe that only checks process PID.

startupProbe:
  exec:
    command: ["cat", "/tmp/ready"]
  failureThreshold: 30
  periodSeconds: 5
livenessProbe:
  exec:
    command: ["pgrep", "-f", "myservice"]
  initialDelaySeconds: 0
  periodSeconds: 30
  failureThreshold: 4

Error: kubelet: Failed to start Container "my-app": OCI runtime create failed: container_linux.go:380: starting container process caused "process_linux.go:449: setting cgroup config for process caused "invalid argument"

Why: The pod exceeded the node’s ephemeral storage limit, causing the container runtime to reject the start. Probe failures cascade because the pod never gets to a running state. Fix: Increase ephemeral-storage limits or clean up old logs. Example:

resources:
  limits:
    ephemeral-storage: "2Gi"
  requests:
    ephemeral-storage: "1Gi"

Error: node pressure: memory logged by kubelet

Why: The node is swapping or OOM‑killing other pods, making the kubelet defer health checks. Fix:

kubectl top node               # spot memory pressure
kubectl describe node <node>   # look for `MemoryPressure=True`

Evict non‑critical pods or add more worker nodes.

Error: iptables: No chain/target/match by that name during probe routing

Why: kube‑proxy failed to install iptables rules, often after a CNI plugin upgrade. Fix: Restart kube‑proxy daemonset with kubectl rollout restart ds/kube-proxy -n kube-system. Verify with iptables -L -t nat that KUBE-POSTROUTING exists.

Monitoring, Alerting, and Preventing Future Issues

Best‑in‑Class Observability Tools for 2025

• Prometheus 2.47 with the kube-state-metrics exporter gives you kube_pod_container_status_* series for probe health.
• Grafana 10.2 dashboards (Kubernetes / Pods / Health) visualize failure trends across clusters.
• OpenTelemetry Collector can ingest probe latency as a custom metric and correlate it with downstream service spans (Linkerd automatically adds linkerd-probe attributes).
• Falco runtime security rules can fire on execve of /usr/local/bin/kubelet when the healthcheck flag is used, alerting on unusually high probe failure rates.

Implementing SLI/SLO‑Based Alerting Frameworks

Define a Service Level Indicator for readiness:

sli: pod_ready_ratio
value: sum(kube_pod_container_status_ready{namespace="prod"}) / count(kube_pod_container_status_ready{namespace="prod"})
target: 0.99

Use Sloth (a declarative SLO tool) to generate Prometheus rules that trigger a warning when the 5‑minute window falls below 0.95, and a critical alert at 0.90. This approach mirrors the LinkedIn study that cut noisey alerts by 70 %.

# sloth.yml – SLO definition
slo:
  name: pod-readiness
  objectives:
  - ratio:
      good: sum(kube_pod_container_status_ready{namespace="prod"})
      total: count(kube_pod_container_status_ready{namespace="prod"})
    target: 0.99
    window: 5m
    alerting:
      - warning: 0.95
      - critical: 0.90

Frequently asked questions

If you’ve run into a stubborn “agent not responding” case that isn’t covered here, drop a comment with your cluster version and probe config. I’m happy to dive into the details and tweak the patterns for your environment. Happy debugging!