Monitoring & Logging JavaScript AI Agents in Production

💡 Pro Tip: Skipping observability until after your AI agent goes live usually ends with painful firefighting sessions.

TL;DR

• Start simple: Add a correlation ID and a few latency gauges before expanding.
• Log with structure: JSON logs let you filter by token usage or LLM call type.
• Metrics matter: Track latency, cost per run, and success‑rate together.
• Trace end‑to‑end: OpenTelemetry + Jaeger shows every step of an agentic workflow.
• Iterate: Use semantic evaluation pipelines to turn raw logs into performance scores.

Before you start, you need:

• Node.js ≥ 18, npm ≥ 9.
• LangChain.js v0.0.150 (or newer).
• OpenTelemetry SDK for JavaScript v1.13.0.
• A Prometheus server reachable from your service.
• Jaeger or Grafana Tempo for distributed tracing.
• Access to an LLM endpoint (e.g., OpenAI v1.0.0 API key).

Introduction: The Observability Challenge for AI Agents

A few weeks ago, the team behind nileshblog.tech rolled out a new travel‑assistant chatbot. Within minutes, users reported bizarre itinerary suggestions that mixed up dates, destinations, and even currencies. The on‑call engineer dug through generic logs, found nothing useful, and spent two hours chasing a phantom bug. When the root cause finally emerged—a mis‑routed LLM call caused by a stale cache key—the incident had already cost the company over $1,200 in unnecessary API usage.

That story illustrates a core truth: traditional monitoring tools, built for request/response services, stumble when faced with agentic workflows that span multiple LLM calls, vector searches, and tool invocations. Observability for AI agents must capture not only where a request traveled, but what the model generated, how many tokens it consumed, and whether the output met business goals.

⚠️ Warning: Relying on error‑rate alerts alone hides performance regressions that silently increase cost.

Architecting for Observability in Node.js

Instrumenting Agent Steps & LLM Interactions

An AI agent is a state machine. Each transition—fetching a knowledge snippet, prompting the LLM, calling an external API—produces a measurable event. To watch those events, wrap every step in an OpenTelemetry span. The span name should describe the operation, for example "LLM:ChatCompletion" or "VectorDB:SimilaritySearch".

// lib/telemetry.js (OpenTelemetry v1.13.0)
import { trace, context, propagation } from '@opentelemetry/api';
import { NodeTracerProvider } from '@opentelemetry/sdk-trace-node';
import { BatchSpanProcessor } from '@opentelemetry/sdk-trace-base';
import { JaegerExporter } from '@opentelemetry/exporter-jaeger';
import { registerInstrumentations } from '@opentelemetry/instrumentation';
import { HttpInstrumentation } from '@opentelemetry/instrumentation-http';

const provider = new NodeTracerProvider();
const exporter = new JaegerExporter({
  endpoint: 'http://jaeger:14268/api/traces',
});
provider.addSpanProcessor(new BatchSpanProcessor(exporter));
provider.register();

registerInstrumentations({
  instrumentations: [new HttpInstrumentation()],
});

export const tracer = trace.getTracer('nileshblog.tech-agent', '1.0.0');
export const propagateCorrelation = (carrier) => {
  propagation.inject(context.active(), carrier);
};

The snippet sets up a tracer that ships spans to Jaeger. Notice the explicit version numbers; they help teammates replicate the exact environment.

When the agent receives a user request, generate a correlation ID and store it in the OpenTelemetry context. Propagate that ID through every downstream call, including HTTP headers to the LLM service.

// lib/agent.js
import { v4 as uuidv4 } from 'uuid';
import { tracer, propagateCorrelation } from './telemetry.js';
import { OpenAI } from 'openai';

const openai = new OpenAI({ apiKey: process.env.OPENAI_API_KEY });

export async function handleMessage(userMessage) {
  const correlationId = uuidv4();
  const rootSpan = tracer.startSpan('Agent:HandleMessage', {
    attributes: { 'correlation.id': correlationId },
  });

  try {
    // Attach correlation to outgoing HTTP calls
    const headers = {};
    propagateCorrelation(headers);
    const response = await openai.chat.completions.create(
      {
        model: 'gpt-4o-mini',
        messages: [{ role: 'user', content: userMessage }],
      },
      { headers }
    );

    // Record token usage as a metric (see later)
    recordTokenMetric(response.usage.total_tokens, correlationId);

    return response.choices[0].message.content;
  } catch (err) {
    rootSpan.recordException(err);
    throw err;
  } finally {
    rootSpan.end();
  }
}

Structured Logging with Context

JSON logs let you index on any field—correlation.id, token_usage, step. Use a logger that writes one line per event, avoiding multiline messages that break parsers.

// lib/logger.js
import pino from 'pino';

export const logger = pino({
  level: process.env.LOG_LEVEL || 'info',
  timestamp: () => `,"time":"${new Date().toISOString()}"`,
  base: null,
  formatters: {
    level(label) {
      return { level: label };
    },
  },
});

A typical log entry from the agent looks like:

{
  "time":"2026-06-28T14:23:07.123Z",
  "level":"info",
  "msg":"LLM request completed",
  "correlation.id":"9f7c2d4e-1b3a-4d9f-8c6a-2e5f9a1b6c99",
  "model":"gpt-4o-mini",
  "prompt_tokens":58,
  "completion_tokens":112,
  "total_tokens":170,
  "latency_ms":842
}

Performance Metrics: Latency, Cost, and Success Rates

Prometheus excels at time‑series data. Expose a /metrics endpoint that emits counters and histograms for:

• Latency per step (histogram_latency_seconds with label step).
• Token usage (counter_total_tokens with label model).
• Success rate (counter_success_total vs counter_total_requests).
• Cost per run (gauge_cost_usd derived from token pricing).

// lib/metrics.js
import client from 'prom-client';
const register = new client.Registry();

export const latencyHistogram = new client.Histogram({
  name: 'agent_step_latency_seconds',
  help: 'Latency of each agent step',
  labelNames: ['step'],
  buckets: [0.05, 0.1, 0.3, 0.5, 1, 2, 5],
});
register.registerMetric(latencyHistogram);

export const tokenCounter = new client.Counter({
  name: 'agent_total_tokens',
  help: 'Total tokens consumed by the agent',
  labelNames: ['model'],
});
register.registerMetric(tokenCounter);

export const successCounter = new client.Counter({
  name: 'agent_success_total',
  help: 'Number of successful agent runs',
});
register.registerMetric(successCounter);

export const requestCounter = new client.Counter({
  name: 'agent_requests_total',
  help: 'All agent invocations',
});
register.registerMetric(requestCounter);

export const costGauge = new client.Gauge({
  name: 'agent_cost_usd',
  help: 'Estimated cost per request in USD',
});
register.registerMetric(costGauge);

export async function metricsHandler(req, res) {
  res.set('Content-Type', register.contentType);
  res.end(await register.metrics());
}

Add the handler to your Express server:

import express from 'express';
import { metricsHandler } from './metrics.js';
import { handleMessage } from './agent.js';

const app = express();
app.use(express.json());

app.post('/chat', async (req, res) => {
  try {
    const reply = await handleMessage(req.body.message);
    res.json({ reply });
  } catch (e) {
    res.status(500).json({ error: e.message });
  }
});

app.get('/metrics', metricsHandler);
app.listen(3000, () => console.log('Server listening on :3000'));

Tracing Agent Workflows for Debugging

With spans and logs aligned on the correlation ID, a single Jaeger trace shows a complete conversation: user request → tool lookup → LLM call → post‑processing. Click any span to view attached logs, latency, and custom attributes like token_usage. This view turns hours of log‑tail hunting into a few seconds of visual inspection.

💡 Pro Tip: Attach the embedding vector summary as a span attribute (embedding.avg_norm) to spot outliers that may indicate hallucinations.

Essential Monitoring Metrics for AI Agents

Operational Metrics: Latency & Throughput

Latency determines user experience. Track p95 latency per step; a sudden spike on the "VectorDB:SimilaritySearch" span often signals an overloaded index or a network hiccup. Throughput (requests per second) correlates with cost, so keep an eye on the ratio between throughput and token usage.

Business & Quality Metrics: Cost per Run & Success Rate

OpenAI charges roughly $0.00015 per 1 k tokens for the gpt-4o-mini model. Multiply the token counter by this rate, feed the result into the agent_cost_usd gauge, and set an alert when average cost exceeds a budgeted threshold.

Success rate measures functional correctness. Define task completion as a boolean returned from downstream validation logic (e.g., does the itinerary contain all required cities?). Increment agent_success_total only on a true outcome.

LLM‑Specific Metrics: Token Usage & Hallucination Indicators

Token counters let you spot runaway prompts. If prompt_tokens grows steadily, your prompt engineering may be drifting. Hallucination detection requires more nuance: compute cosine similarity between the LLM’s response embedding and a set of vetted reference embeddings. Log a hallucination_score attribute; set an alert if the score falls below a configurable floor (e.g., 0.78).

My take: I treat the hallucination score as a first‑class metric, not an after‑thought. When the score slipped, I caught a downstream bug that inserted an extra newline into the prompt template—something that would have been invisible in plain logs.

Advanced Techniques: From Reactive to Proactive

Evaluating Performance with Semantic Similarity & Guardrails

Build a golden dataset of representative user queries and expected intents. After each deployment, run those queries through the live agent, embed the responses with OpenAI’s text-embedding-3-large (v1.0.0), and compute cosine similarity against the gold embeddings.

import { OpenAI } from 'openai';
import { cosineSimilarity } from './utils.js';
const openai = new OpenAI({ apiKey: process.env.OPENAI_API_KEY });

export async function evaluateSemanticScore(prompt, goldenEmbedding) {
  const response = await openai.chat.completions.create({
    model: 'gpt-4o-mini',
    messages: [{ role: 'user', content: prompt }],
  });
  const embedding = await openai.embeddings.create({
    model: 'text-embedding-3-large',
    input: response.choices[0].message.content,
  });
  return cosineSimilarity(embedding.data[0].embedding, goldenEmbedding);
}

Store the resulting scores in Prometheus (gauge_semantic_score) and plot trends. A downward drift signals regression before users notice.

Implementing Canary Deployments & A/B Testing for Agents

Deploy a new version of the agent to 10 % of traffic using a feature flag. Tag the incoming request with a deployment_id attribute on the root span. Compare latency, cost, success‑rate, and semantic‑score across the two groups. Grafana can overlay the two time series, letting you decide whether to promote the canary.

Anomaly Detection in Conversational Logs

Leverage unsupervised clustering on embeddings of recent chats. When a cluster’s centroid shifts beyond a statistical threshold, raise an anomaly alert. This technique catches sudden changes in user intent distribution (e.g., a holiday‑season surge) that may require scaling vector stores or adjusting prompt temperature.

Real‑World Implementation with OpenTelemetry

Code Example: Instrumenting a LangChain.js Agent

LangChain.js abstracts chain steps into Runnables. Wrap each runnable with an OpenTelemetry span using a helper function.

// lib/langchain-otel.js
import { tracer } from './telemetry.js';
import { RunnableSequence } from 'langchain/runnables';

/**
 * Wraps a LangChain Runnable with an OTEL span.
 * @param {Runnable} runnable - The original LangChain component.
 * @param {string} name - Human‑readable span name.
 * @returns {Runnable}
 */
export function instrumentRunnable(runnable, name) {
  return runnable.withConfig({
    callbacks: [
      {
        async handleStart() {
          this._span = tracer.startSpan(name);
        },
        async handleEnd(output) {
          this._span.setAttribute('output.length', output?.length ?? 0);
          this._span.end();
          return output;
        },
        async handleError(error) {
          this._span.recordException(error);
          this._span.setStatus({ code: 2, message: error.message });
          this._span.end();
        },
      },
    ],
  });
}

// Example usage in nileshblog.tech chatbot
import { ChatOpenAI } from 'langchain/chat_models/openai';
import { PromptTemplate } from 'langchain/prompts';
import { LLMChain } from 'langchain/chains';
import { instrumentRunnable } from './langchain-otel.js';

const chatModel = new ChatOpenAI({ modelName: 'gpt-4o-mini', temperature: 0.2 });
const prompt = PromptTemplate.fromTemplate('Summarize: {input}');
const chain = new LLMChain({ llm: chatModel, prompt });

const tracedChain = instrumentRunnable(chain, 'LLMChain:Summarize');

export async function summarize(text) {
  try {
    const result = await tracedChain.invoke({ input: text });
    return result.text;
  } catch (err) {
    // Propagate error after logging
    console.error('Summarization failed', err);
    throw err;
  }
}

The code adds a span for the entire chain, automatically recording start, end, and any exception. Because LangChain’s callback system already supports async hooks, the instrumentation stays lightweight—overhead stays under 5 % in typical workloads.

Visualizing Traces in Jaeger or Grafana

After deploying the instrumented service, open Jaeger at http://localhost:16686. Search by correlation.id to see a single user’s journey. In Grafana, add a Explore panel that queries Prometheus for agent_step_latency_seconds_bucket{step="LLM:ChatCompletion"} and overlays the histogram with the semantic‑score gauge.

graph TD
  A[User Request] -->|HTTP| B[Ingress Proxy];
  B -->|Start Span| C[Agent:HandleMessage];
  C --> D[VectorDB:SimilaritySearch];
  D --> E[LLM:ChatCompletion];
  E --> F[Post‑Processing];
  F -->|End Span| G[Response Sent];
  style A fill:#f9f,stroke:#333,stroke-width:2px
  style G fill:#bbf,stroke:#333,stroke-width:2px

The diagram shows the flow of spans and how each component contributes to the overall trace.

Correlating Logs, Metrics, and Traces

Because every span carries the correlation.id, you can join data across three stores:

Having a single identifier eliminates the “where did that token count come from?” mystery.

Case Studies & Trade‑offs

Trade‑off: Granularity vs. Performance & Cost

Fine‑grained spans (e.g., one per token) provide spectacular detail but increase CPU usage and network traffic to the collector. A pragmatic rule of thumb: span per external call and span per major internal step. Keep internal loops inside a single span and record aggregate counters for inner iterations.

Example: Scaling Agent Monitoring for an E‑commerce Chatbot

nileshblog.tech recently powered an e‑commerce chatbot handling 4 k RPS during a flash sale. The team:

1. Set the OpenTelemetry exporter to batch mode with a 5‑second flush interval, reducing overhead by 12 %.
2. Switched log storage to Elastic Cloud with an index‑time pipeline that drops debug‑level entries for non‑production traffic.
3. Introduced a Prometheus relabel rule that aggregates token counters per minute, keeping the series count under 10 k.

Result: monitoring latency stayed under 80 ms per request, cost per request dropped 18 % after spotting an off‑by‑one error in a price‑lookup tool.

Common Errors & Fixes

• Missing correlation ID in downstream services – Ensure you call propagateCorrelation(headers) before each HTTP request, and that the downstream service extracts it using propagation.extract.
• Exporter overload – If Jaeger shows “queue full” warnings, increase the batch size or enable back‑pressure handling via OTEL_EXPORTER_OTLP_BATCH_MAX_QUEUE_SIZE.
• Metric duplication – Register each Prometheus metric once; attempting to re‑register throws Error: Duplicate metric name. Place metric definitions in a singleton module.
• Embedding latency spikes – Cache frequently used embeddings; a simple LRU cache (lru-cache@7.14.0) can cut embedding latency by >30 %.
• High memory usage in tracing – Limit the number of attributes per span; large payloads (full LLM responses) belong in logs, not span attributes.

CTA

If this guide helped you tighten observability for your AI agents, share your experiences in the comments below. Follow the blog at nileshblog.tech for more deep dives, and consider subscribing to get the latest system‑design patterns straight to your inbox.

FAQs

What’s the most important metric to track for a production AI agent initially?

Success Rate (or Task Completion Rate) combined with Average Latency give you a direct view of reliability and user experience. After those stabilize, add Cost per Task to keep budgets in check.

How do I trace a single user session through a multi‑step AI agent workflow?

Generate a unique correlation ID (e.g., a UUID) when the session starts, then propagate it via OpenTelemetry’s Context API. Attach the same ID to every log line, metric label, and span attribute. Jaeger’s search bar can filter on that ID to display the whole trace.

What’s a simple way to evaluate if my agent’s responses are improving over time?

Build a semantic evaluation pipeline: keep a set of “golden” queries with expected intents, run them after each deployment, embed the new responses, and compare cosine similarity against the golden embeddings. Track the average similarity as a gauge metric.

Author Bio:
I’m Nilesh Raut, a Software Development Engineer with 2+ years of experience, specializing in Go, JavaScript, Python, Docker, Kubernetes, Git, Jenkins, microservices, and system design (LLD/HLD), backed by a strong foundation in data structures and algorithms. Alongside my engineering journey, I bring 4+ years of hands‑on experience in SEO, where I’ve worked extensively on content strategy, keyword research, technical SEO, and organic growth, helping products and businesses scale efficiently by aligning solid technology with search‑driven performance.