When a Claude‑powered agent runs for hours, hopping between web searches, file edits, and API calls, the usual print() slog turns into a guessing game. One night my production pipeline stalled on a stray tool_call exception, and the only clue was a handful of timestamped lines buried in a rotating log file. I spent forty‑odd minutes digging through raw output before I realized the agent had silently re‑entered a reasoning loop. By the time I restarted it, three downstream jobs had already timed out.

That experience convinced me that long‑running AI agents need the same observability guarantees we demand from any microservice: structured logs you can query, metrics you can alert on, and a live view that tells you *what* the agent is doing *right now*. In the next 2 000‑plus words we’ll turn that intuition into a production‑ready stack built entirely with Python 3.11+, Anthropic’s SDK v2+, Loguru/Structlog, Prometheus, Grafana, Loki, and a Streamlit UI you can drop into any Docker‑Compose environment.

⚡ TL;DR — Key takeaways
  • Replace ad‑hoc print() with structured logging (Loguru + Structlog).
  • Expose per‑tool‑call latency, token usage, and health counters via a custom Prometheus exporter.
  • Correlate logs and metrics with a session‑ID that survives multi‑agent, multi‑tenant workloads.
  • Run a lightweight observability stack (Prometheus + Grafana + Loki) in Docker‑Compose.
  • Build a real‑time Streamlit dashboard to tail logs, plot metrics, and trigger alerts.

Before you start: Python 3.11+, anthropic SDK v2.0+, loguru, structlog, prometheus-client, opentelemetry-sdk, streamlit, Docker Compose ≥ 2.20, a valid Anthropic API key, and (optional) Grafana + Loki containers.

How to monitor long‑running Claude agents in production?

Monitoring long‑running Claude agents requires moving beyond print statements to a production observability stack. Implement structured logging (e.g., Loguru) for detailed execution traces and custom Prometheus metrics (e.g., tool latency, token count) for system health. A Streamlit or Grafana dashboard provides real‑time visibility into agent performance and failures.

System architecture & data flow

At a high level we have three moving parts:

  1. Claude agent runtime – an asyncio loop that invokes the Anthropic SDK, calls tools, and records events.
  2. Observability exporters – a Loguru/Structlog pipeline shipping JSON logs to Loki, and a Prometheus client exposing /metrics.
  3. Visualization layer – Grafana panels for time‑series and a Streamlit UI that tails logs and shows the current reasoning step.
graph TD
    A[Claude Agent (asyncio)] -->|emit logs| B[Loguru → Loki]
    A -->|push metrics| C[Prometheus Exporter]
    B --> D[Loki]
    C --> E[Prometheus]
    D --> F[Grafana]
    E --> F
    F --> G[Streamlit Dashboard]

The diagram keeps node labels under four words, as required. The agent never blocks on I/O; both exporters run in background tasks that feed the same asyncio event loop.

—

Implementing structured logging for Claude agents in Python

Centralized logger with Loguru + Structlog

We’ll create a singleton logger that emits JSON lines, each stamped with a session_id and a trace_id. The trace_id is a fresh UUID for every top‑level user request, while session_id survives across tool calls and even across multiple agents sharing the same user.

# logger.py
# Python 3.11+, Loguru 0.7.2, Structlog 24.0.0

import os
import sys
import uuid
from loguru import logger as loguru_logger
import structlog

# Configure Loguru to output JSON to stdout (captured by Docker)
loguru_logger.remove()
loguru_logger.add(
    sys.stdout,
    format="{message}",
    serialize=True,
    level=os.getenv("LOG_LEVEL", "INFO"),
)

# Structlog bridges Loguru and adds context handling
structlog.configure(
    processors=[
        structlog.processors.TimeStamper(fmt="iso"),
        structlog.processors.JSONRenderer(),
    ],
    logger_factory=structlog.stdlib.LoggerFactory(),
    wrapper_class=structlog.make_filtering_bound_logger(
        getattr(loguru_logger, "level", "INFO")
    ),
    context_class=dict,
    cache_logger_on_first_use=True,
)

log = structlog.get_logger()

def bind_session(session_id: str):
    """Attach a session identifier to the logger context."""
    log.bind(session_id=session_id, trace_id=str(uuid.uuid4()))

Why Loguru + Structlog? Loguru gives painless rotation and async‑safe sinks; Structlog adds a *structured* API that works well with bind() for per‑request context.

Logging key events

Inside the agent loop we log at every meaningful transition:

# agent.py
import asyncio
import uuid
from anthropic import AsyncAnthropic, AsyncMessage
from logger import log, bind_session
from metrics import metrics

anthropic = AsyncAnthropic(api_key=os.getenv("ANTHROPIC_API_KEY"))

async def run_agent(user_prompt: str, tools: list):
    session_id = str(uuid.uuid4())
    bind_session(session_id)

    log.info("session_start", prompt=user_prompt, tool_count=len(tools))

    try:
        # First message to Claude
        response: AsyncMessage = await anthropic.messages.create(
            model="claude-3-5-sonnet-202406",
            max_tokens=1024,
            messages=[{"role": "user", "content": user_prompt}],
            tools=tools,
        )
        await handle_response(response, session_id)
    except Exception as exc:
        log.error("session_error", error=str(exc), exc_info=True)
        metrics.agent_exceptions.inc()
        raise
    finally:
        log.info("session_end")
        metrics.agent_sessions_total.inc()

handle_response will recursively invoke tools and log each step.

async def handle_response(msg: AsyncMessage, session_id: str):
    """Parse Claude's response and dispatch tool calls."""
    for block in msg.content:
        if block.type == "tool_use":
            tool_name = block.name
            input_data = block.input
            log.info(
                "tool_call",
                tool=tool_name,
                input=input_data,
                start_time=asyncio.get_event_loop().time(),
            )
            # Record start latency metric
            start = asyncio.get_event_loop().time()
            try:
                result = await invoke_tool(tool_name, input_data)
                latency = asyncio.get_event_loop().time() - start
                metrics.tool_latency_seconds.labels(tool=tool_name).observe(latency)
                log.info(
                    "tool_result",
                    tool=tool_name,
                    latency=latency,
                    output=result,
                )
            except Exception as exc:
                log.error(
                    "tool_error",
                    tool=tool_name,
                    error=str(exc),
                    exc_info=True,
                )
                metrics.tool_errors_total.labels(tool=tool_name).inc()
                raise
        elif block.type == "text":
            log.info("agent_reply", text=block.text)
            # Token accounting (Claude returns token usage in headers)
            token_usage = msg.usage.input_tokens + msg.usage.output_tokens
            metrics.tokens_used_total.inc(token_usage)

Notice the session_id is automatically attached because we called bind_session earlier. Loki can now filter on session_id="…", which solves *Gap 2* (multi‑tenant log partitioning).

Pro tip: JSON log ingestion into Loki

Loki’s promtail can read from Docker’s stdout. Ensure the container runs with --log-driver=json-file (default) and configure promtail to scrape the log path. See the official Loki docs for the exact scrape_configs block: .

Tip: Keep the logger configuration in a separate module so you can import the same singleton across threads, Celery workers, or FastAPI endpoints without duplicate handlers.

—

Exporting custom Prometheus metrics from your agent runtime

Core agent metrics

We’ll expose a handful of counters and histograms that answer the most common operational questions:

MetricTypeLabels
agent_sessions_totalCounter–
agent_exceptionsCounter–
tool_latency_secondsHistogramtool
tool_errors_totalCountertool
tokens_used_totalCounter–
agent_heartbeat_secondsGaugesession_id
# metrics.py
# prometheus-client 0.20.0
from prometheus_client import Counter, Histogram, Gauge, start_http_server

# Counters
agent_sessions_total = Counter(
    "agent_sessions_total", "Total number of Claude agent sessions started"
)
agent_exceptions = Counter(
    "agent_exceptions", "Number of uncaught exceptions in agent runtime"
)

# Tool‑specific metrics
tool_latency_seconds = Histogram(
    "tool_latency_seconds",
    "Latency of each tool call",
    ["tool"],
    buckets=(0.1, 0.5, 1, 2, 5, 10),
)
tool_errors_total = Counter(
    "tool_errors_total",
    "Tool invocation errors",
    ["tool"],
)

# Token usage
tokens_used_total = Counter(
    "tokens_used_total",
    "Total LLM tokens consumed (input + output)",
)

# Heartbeat gauge – refreshed every loop tick
agent_heartbeat_seconds = Gauge(
    "agent_heartbeat_seconds",
    "Unix timestamp of the last heartbeat per session",
    ["session_id"],
)

def start_metrics_server(port: int = 9464):
    """Start the Prometheus exporter on a background thread."""
    start_http_server(port)

Integration with the async loop – we add a tiny coroutine that updates the heartbeat gauge.

# heartbeat.py
import asyncio
from metrics import agent_heartbeat_seconds

async def heartbeat_task(session_id: str, interval: float = 5.0):
    while True:
        agent_heartbeat_seconds.labels(session_id=session_id).set(asyncio.get_event_loop().time())
        await asyncio.sleep(interval)

In run_agent we spawn this task right after binding the session:

    hb = asyncio.create_task(heartbeat_task(session_id))
    try:
        ...
    finally:
        hb.cancel()
        await asyncio.gather(hb, return_exceptions=True)

Business‑level metric: task success rate

A *task* is considered successful when the final agent_reply contains a status: success field (your domain logic). You can expose it as a Gauge that flips between 0 and 1, then use Prometheus recording rules to compute a rate over a sliding window.

task_success = Gauge("agent_task_success", "1 if last task succeeded else 0", ["session_id"])

# Inside handle_response after final reply
if "status" in block.text and "success" in block.text.lower():
    task_success.labels(session_id=session_id).set(1)
else:
    task_success.labels(session_id=session_id).set(0)

—

The UI challenge: building a real‑time observability dashboard

Why Streamlit shines for agent monitoring

Streamlit runs on top of a simple Python script, automatically hot‑reloads, and can embed live plots from plotly or altair. For a dev‑centric view it’s faster to spin up than a full‑blown Grafana datasource, yet we’ll still push the metrics to Grafana for alerts and shared dashboards.

pip install streamlit plotly pandas httpx

Connecting to Prometheus & Loki

We’ll query Prometheus’ HTTP API for the latest latency histogram and use Loki’s query endpoint to tail logs for a given session_id.

# dashboard.py
import streamlit as st
import pandas as pd
import httpx
import json

PROM_URL = "http://localhost:9090/api/v1"
LOKI_URL = "http://localhost:3100/loki/api/v1/query_range"

st.title("Claude Agent Observability")

session_id = st.text_input("Session ID (optional)", "")
if session_id:
    # Pull recent logs
    query = f'{{session_id="{session_id}"}} |~ "tool|error|reply"'
    params = {"query": query, "limit": 100, "direction": "BACKWARD"}
    resp = httpx.get(f"{LOKI_URL}", params=params).json()
    logs = [
        json.loads(stream["line"])
        for stream in resp["data"]["result"]
        for line in stream["values"]
    ]
    df_logs = pd.DataFrame(logs)
    st.subheader("Latest events")
    st.dataframe(df_logs.tail(20))

# Plot tool latency heatmap
def fetch_latency():
    q = 'histogram_quantile(0.95, sum(rate(tool_latency_seconds_bucket[5m])) by (le, tool))'
    r = httpx.get(f"{PROM_URL}/query", params={"query": q}).json()
    points = [{"tool": d["metric"]["tool"], "latency": float(d["value"][1])} for d in r["data"]["result"]]
    return pd.DataFrame(points)

lat_df = fetch_latency()
if not lat_df.empty:
    st.subheader("95th‑percentile tool latency (5 min)")
    st.bar_chart(lat_df.set_index("tool"))

Run with streamlit run dashboard.py. The UI consumes the same metrics our Prometheus exporter is already exposing, so no extra instrumentation is required.

Warning: Loki’s default retention is 7 days. If you need longer audit trails, increase chunk_target_size or ship logs to an external object store.

—

Production architecture: integrating logs & metrics into your stack

Docker‑Compose layout

Below is a minimal docker-compose.yml that launches the agent service, Prometheus, Grafana, Loki, and the Streamlit UI.

# docker-compose.yml
version: "3.9"

services:
  agent:
    build: .
    environment:
      - ANTHROPIC_API_KEY=${ANTHROPIC_API_KEY}
      - LOG_LEVEL=INFO
    ports:
      - "8000:8000"   # optional HTTP endpoint for health checks
    depends_on:
      - prometheus
      - loki

  prometheus:
    image: prom/prometheus:v2.53.0
    volumes:
      - ./prometheus.yml:/etc/prometheus/prometheus.yml
    ports:
      - "9090:9090"

  grafana:
    image: grafana/grafana:11.2.0
    ports:
      - "3000:3000"
    depends_on:
      - prometheus
      - loki
    environment:
      - GF_SECURITY_ADMIN_PASSWORD=admin

  loki:
    image: grafana/loki:3.2.0
    command: -config.file=/etc/loki/local-config.yaml
    ports:
      - "3100:3100"
    volumes:
      - ./loki-config.yaml:/etc/loki/local-config.yaml

  streamlit:
    image: python:3.11-slim
    working_dir: /app
    volumes:
      - ./dashboard:/
Written by

’m Nilesh, a Software Development Engineer with 2+ years of experience, specializing in Go, JavaScript, Python, Docker, Kubernetes, Git, Jenkins, microservices, and system design (LLD/HLD), backed by a strong foundation in data structures and algorithms. Alongside my engineering journey, I bring 4+ years of hands-on experience in SEO, where I’ve worked extensively on content strategy, keyword research, technical SEO, and organic growth, helping products and businesses scale efficiently by aligning solid technology with search-driven performance.