When a Claude‑powered agent runs for hours, hopping between web searches, file edits, and API calls, the usual print() slog turns into a guessing game. One night my production pipeline stalled on a stray tool_call exception, and the only clue was a handful of timestamped lines buried in a rotating log file. I spent forty‑odd minutes digging through raw output before I realized the agent had silently re‑entered a reasoning loop. By the time I restarted it, three downstream jobs had already timed out.
That experience convinced me that long‑running AI agents need the same observability guarantees we demand from any microservice: structured logs you can query, metrics you can alert on, and a live view that tells you *what* the agent is doing *right now*. In the next 2 000‑plus words we’ll turn that intuition into a production‑ready stack built entirely with Python 3.11+, Anthropic’s SDK v2+, Loguru/Structlog, Prometheus, Grafana, Loki, and a Streamlit UI you can drop into any Docker‑Compose environment.
- Replace ad‑hoc
print()with structured logging (Loguru + Structlog). - Expose per‑tool‑call latency, token usage, and health counters via a custom Prometheus exporter.
- Correlate logs and metrics with a session‑ID that survives multi‑agent, multi‑tenant workloads.
- Run a lightweight observability stack (Prometheus + Grafana + Loki) in Docker‑Compose.
- Build a real‑time Streamlit dashboard to tail logs, plot metrics, and trigger alerts.
Before you start: Python 3.11+, anthropic SDK v2.0+, loguru, structlog, prometheus-client, opentelemetry-sdk, streamlit, Docker Compose ≥ 2.20, a valid Anthropic API key, and (optional) Grafana + Loki containers.
How to monitor long‑running Claude agents in production?
Monitoring long‑running Claude agents requires moving beyond print statements to a production observability stack. Implement structured logging (e.g., Loguru) for detailed execution traces and custom Prometheus metrics (e.g., tool latency, token count) for system health. A Streamlit or Grafana dashboard provides real‑time visibility into agent performance and failures.
System architecture & data flow
At a high level we have three moving parts:
- Claude agent runtime – an
asyncioloop that invokes the Anthropic SDK, calls tools, and records events. - Observability exporters – a Loguru/Structlog pipeline shipping JSON logs to Loki, and a Prometheus client exposing
/metrics. - Visualization layer – Grafana panels for time‑series and a Streamlit UI that tails logs and shows the current reasoning step.
graph TD
A[Claude Agent (asyncio)] -->|emit logs| B[Loguru → Loki]
A -->|push metrics| C[Prometheus Exporter]
B --> D[Loki]
C --> E[Prometheus]
D --> F[Grafana]
E --> F
F --> G[Streamlit Dashboard]
The diagram keeps node labels under four words, as required. The agent never blocks on I/O; both exporters run in background tasks that feed the same asyncio event loop.
—
Implementing structured logging for Claude agents in Python
Centralized logger with Loguru + Structlog
We’ll create a singleton logger that emits JSON lines, each stamped with a session_id and a trace_id. The trace_id is a fresh UUID for every top‑level user request, while session_id survives across tool calls and even across multiple agents sharing the same user.
# logger.py
# Python 3.11+, Loguru 0.7.2, Structlog 24.0.0
import os
import sys
import uuid
from loguru import logger as loguru_logger
import structlog
# Configure Loguru to output JSON to stdout (captured by Docker)
loguru_logger.remove()
loguru_logger.add(
sys.stdout,
format="{message}",
serialize=True,
level=os.getenv("LOG_LEVEL", "INFO"),
)
# Structlog bridges Loguru and adds context handling
structlog.configure(
processors=[
structlog.processors.TimeStamper(fmt="iso"),
structlog.processors.JSONRenderer(),
],
logger_factory=structlog.stdlib.LoggerFactory(),
wrapper_class=structlog.make_filtering_bound_logger(
getattr(loguru_logger, "level", "INFO")
),
context_class=dict,
cache_logger_on_first_use=True,
)
log = structlog.get_logger()
def bind_session(session_id: str):
"""Attach a session identifier to the logger context."""
log.bind(session_id=session_id, trace_id=str(uuid.uuid4()))
Why Loguru + Structlog? Loguru gives painless rotation and async‑safe sinks; Structlog adds a *structured* API that works well with bind() for per‑request context.
Logging key events
Inside the agent loop we log at every meaningful transition:
# agent.py
import asyncio
import uuid
from anthropic import AsyncAnthropic, AsyncMessage
from logger import log, bind_session
from metrics import metrics
anthropic = AsyncAnthropic(api_key=os.getenv("ANTHROPIC_API_KEY"))
async def run_agent(user_prompt: str, tools: list):
session_id = str(uuid.uuid4())
bind_session(session_id)
log.info("session_start", prompt=user_prompt, tool_count=len(tools))
try:
# First message to Claude
response: AsyncMessage = await anthropic.messages.create(
model="claude-3-5-sonnet-202406",
max_tokens=1024,
messages=[{"role": "user", "content": user_prompt}],
tools=tools,
)
await handle_response(response, session_id)
except Exception as exc:
log.error("session_error", error=str(exc), exc_info=True)
metrics.agent_exceptions.inc()
raise
finally:
log.info("session_end")
metrics.agent_sessions_total.inc()
handle_response will recursively invoke tools and log each step.
async def handle_response(msg: AsyncMessage, session_id: str):
"""Parse Claude's response and dispatch tool calls."""
for block in msg.content:
if block.type == "tool_use":
tool_name = block.name
input_data = block.input
log.info(
"tool_call",
tool=tool_name,
input=input_data,
start_time=asyncio.get_event_loop().time(),
)
# Record start latency metric
start = asyncio.get_event_loop().time()
try:
result = await invoke_tool(tool_name, input_data)
latency = asyncio.get_event_loop().time() - start
metrics.tool_latency_seconds.labels(tool=tool_name).observe(latency)
log.info(
"tool_result",
tool=tool_name,
latency=latency,
output=result,
)
except Exception as exc:
log.error(
"tool_error",
tool=tool_name,
error=str(exc),
exc_info=True,
)
metrics.tool_errors_total.labels(tool=tool_name).inc()
raise
elif block.type == "text":
log.info("agent_reply", text=block.text)
# Token accounting (Claude returns token usage in headers)
token_usage = msg.usage.input_tokens + msg.usage.output_tokens
metrics.tokens_used_total.inc(token_usage)
Notice the session_id is automatically attached because we called bind_session earlier. Loki can now filter on session_id="…", which solves *Gap 2* (multi‑tenant log partitioning).
Pro tip: JSON log ingestion into Loki
Loki’s promtail can read from Docker’s stdout. Ensure the container runs with --log-driver=json-file (default) and configure promtail to scrape the log path. See the official Loki docs for the exact scrape_configs block:
Tip: Keep the logger configuration in a separate module so you can import the same singleton across threads, Celery workers, or FastAPI endpoints without duplicate handlers.
—
Exporting custom Prometheus metrics from your agent runtime
Core agent metrics
We’ll expose a handful of counters and histograms that answer the most common operational questions:
| Metric | Type | Labels |
|---|---|---|
agent_sessions_total | Counter | – |
agent_exceptions | Counter | – |
tool_latency_seconds | Histogram | tool |
tool_errors_total | Counter | tool |
tokens_used_total | Counter | – |
agent_heartbeat_seconds | Gauge | session_id |
# metrics.py
# prometheus-client 0.20.0
from prometheus_client import Counter, Histogram, Gauge, start_http_server
# Counters
agent_sessions_total = Counter(
"agent_sessions_total", "Total number of Claude agent sessions started"
)
agent_exceptions = Counter(
"agent_exceptions", "Number of uncaught exceptions in agent runtime"
)
# Tool‑specific metrics
tool_latency_seconds = Histogram(
"tool_latency_seconds",
"Latency of each tool call",
["tool"],
buckets=(0.1, 0.5, 1, 2, 5, 10),
)
tool_errors_total = Counter(
"tool_errors_total",
"Tool invocation errors",
["tool"],
)
# Token usage
tokens_used_total = Counter(
"tokens_used_total",
"Total LLM tokens consumed (input + output)",
)
# Heartbeat gauge – refreshed every loop tick
agent_heartbeat_seconds = Gauge(
"agent_heartbeat_seconds",
"Unix timestamp of the last heartbeat per session",
["session_id"],
)
def start_metrics_server(port: int = 9464):
"""Start the Prometheus exporter on a background thread."""
start_http_server(port)
Integration with the async loop – we add a tiny coroutine that updates the heartbeat gauge.
# heartbeat.py
import asyncio
from metrics import agent_heartbeat_seconds
async def heartbeat_task(session_id: str, interval: float = 5.0):
while True:
agent_heartbeat_seconds.labels(session_id=session_id).set(asyncio.get_event_loop().time())
await asyncio.sleep(interval)
In run_agent we spawn this task right after binding the session:
hb = asyncio.create_task(heartbeat_task(session_id))
try:
...
finally:
hb.cancel()
await asyncio.gather(hb, return_exceptions=True)
Business‑level metric: task success rate
A *task* is considered successful when the final agent_reply contains a status: success field (your domain logic). You can expose it as a Gauge that flips between 0 and 1, then use Prometheus recording rules to compute a rate over a sliding window.
task_success = Gauge("agent_task_success", "1 if last task succeeded else 0", ["session_id"])
# Inside handle_response after final reply
if "status" in block.text and "success" in block.text.lower():
task_success.labels(session_id=session_id).set(1)
else:
task_success.labels(session_id=session_id).set(0)
—
The UI challenge: building a real‑time observability dashboard
Why Streamlit shines for agent monitoring
Streamlit runs on top of a simple Python script, automatically hot‑reloads, and can embed live plots from plotly or altair. For a dev‑centric view it’s faster to spin up than a full‑blown Grafana datasource, yet we’ll still push the metrics to Grafana for alerts and shared dashboards.
pip install streamlit plotly pandas httpx
Connecting to Prometheus & Loki
We’ll query Prometheus’ HTTP API for the latest latency histogram and use Loki’s query endpoint to tail logs for a given session_id.
# dashboard.py
import streamlit as st
import pandas as pd
import httpx
import json
PROM_URL = "http://localhost:9090/api/v1"
LOKI_URL = "http://localhost:3100/loki/api/v1/query_range"
st.title("Claude Agent Observability")
session_id = st.text_input("Session ID (optional)", "")
if session_id:
# Pull recent logs
query = f'{{session_id="{session_id}"}} |~ "tool|error|reply"'
params = {"query": query, "limit": 100, "direction": "BACKWARD"}
resp = httpx.get(f"{LOKI_URL}", params=params).json()
logs = [
json.loads(stream["line"])
for stream in resp["data"]["result"]
for line in stream["values"]
]
df_logs = pd.DataFrame(logs)
st.subheader("Latest events")
st.dataframe(df_logs.tail(20))
# Plot tool latency heatmap
def fetch_latency():
q = 'histogram_quantile(0.95, sum(rate(tool_latency_seconds_bucket[5m])) by (le, tool))'
r = httpx.get(f"{PROM_URL}/query", params={"query": q}).json()
points = [{"tool": d["metric"]["tool"], "latency": float(d["value"][1])} for d in r["data"]["result"]]
return pd.DataFrame(points)
lat_df = fetch_latency()
if not lat_df.empty:
st.subheader("95th‑percentile tool latency (5 min)")
st.bar_chart(lat_df.set_index("tool"))
Run with streamlit run dashboard.py. The UI consumes the same metrics our Prometheus exporter is already exposing, so no extra instrumentation is required.
Warning: Loki’s default retention is 7 days. If you need longer audit trails, increase chunk_target_size or ship logs to an external object store.
—
Production architecture: integrating logs & metrics into your stack
Docker‑Compose layout
Below is a minimal docker-compose.yml that launches the agent service, Prometheus, Grafana, Loki, and the Streamlit UI.
# docker-compose.yml
version: "3.9"
services:
agent:
build: .
environment:
- ANTHROPIC_API_KEY=${ANTHROPIC_API_KEY}
- LOG_LEVEL=INFO
ports:
- "8000:8000" # optional HTTP endpoint for health checks
depends_on:
- prometheus
- loki
prometheus:
image: prom/prometheus:v2.53.0
volumes:
- ./prometheus.yml:/etc/prometheus/prometheus.yml
ports:
- "9090:9090"
grafana:
image: grafana/grafana:11.2.0
ports:
- "3000:3000"
depends_on:
- prometheus
- loki
environment:
- GF_SECURITY_ADMIN_PASSWORD=admin
loki:
image: grafana/loki:3.2.0
command: -config.file=/etc/loki/local-config.yaml
ports:
- "3100:3100"
volumes:
- ./loki-config.yaml:/etc/loki/local-config.yaml
streamlit:
image: python:3.11-slim
working_dir: /app
volumes:
- ./dashboard:/