Your agent calls a payment tool, the API times out after 28 seconds, and your retry logic kicks in. The second attempt succeeds, but the customer’s card gets charged twice. This happens because most agent tool retry exponential backoff idempotency implementations treat LLM tool calls like simple HTTP requests. They ignore the side effects. After reading this, you will implement a retry pattern that survives API rate limits, network blips, and latency spikes without creating duplicate charges, emails, or database records.
Table of Contents
Why do agent tools need retry logic with idempotency?
Agent tool retry with exponential backoff and idempotency keys combines three patterns: exponential wait intervals with jitter to handle transient API failures, unique client-generated keys to prevent duplicate side effects, and persistent storage to track execution state across retries. This prevents duplicate charges, notifications, or data corruption when LLM APIs timeout or return 5xx errors.
LLM agents face failure modes that standard microservices do not. OpenAI’s API maintains a 99.9% uptime SLA but experiences 10-30 second latency spikes that trip default timeouts [platform.openai.com]. Anthropic documents similar transient overload conditions requiring client-side retry [docs.anthropic.com]. When an agent tool times out, the agent framework cannot distinguish between “the request never reached the server” and “the server processed it but the response was lost.” Retrying without protection creates duplicate side effects.
An idempotency key is a unique client-generated token included in API requests. The server uses it to recognize duplicate submissions and return the original response without re-executing side effects. This pattern is essential for safe retries in payment processing and agent tool calls. Without it, you achieve at-most-once delivery (operations may be lost) or at-least-once execution (operations may repeat). Idempotency keys provide exactly-once semantics, guaranteeing single execution for payments, notifications, and stateful operations.
The three failure modes: transport, API, and execution
Transport failures occur before the request reaches the target service. Connection timeouts, DNS resolution failures, and network partitions fall into this category. The operation never executed, so retry is safe. API failures happen when the provider returns an error response. You should retry 500/502/503/504 errors from OpenAI, Anthropic, and similar providers because these are typically transient infrastructure issues. Do NOT retry 400 (bad request) or 401 (unauthorized) as these indicate client-side problems requiring code changes. Execution failures happen after the tool starts running. A database write completes, but the response times out before reaching your agent. This is where retries become dangerous.
When retries become dangerous without idempotency keys
Picture a customer support agent that issues refunds. The refund tool calls Stripe, the payment processor experiences a 529 overloaded error, and your retry logic fires. Without an idempotency key, Stripe processes the refund twice. The customer receives $200 instead of $100. The same risk applies to sending emails, creating database records, and triggering webhooks. Any tool with side effects needs idempotency protection before you enable retry logic.
For more background on handling transient failures in AI systems, see our guide on implementing an exponential backoff retry strategy that avoids making latency worse.
How does exponential backoff work for agent tool calls?
Exponential backoff increases wait time between retries using the formula base_delay * 2^attempt, staying under rate limit windows while allowing temporary congestion to clear. Adding random jitter prevents synchronized client retries that could trigger new throttling. For agent tools, start with shorter initial delays (0.5-1s) since API errors are often transient, but allow longer maximum delays (60-120s) due to cold-start latency on embeddable tools.
The tenacity library provides production-ready wait strategies for this pattern [tenacity.readthedocs.io]. You can chain exponential backoff with jitter using wait_exponential_jitter. This approach works better than fixed delays because it adapts to system conditions. When OpenAI returns a 429 with a retry-after header, you should respect that value instead of calculating your own delay.
Choosing base delay and max retries for LLM APIs
Start with multiplier=1 and min=1 and max=60 for most LLM tool calls. This gives you waits of 1s, 2s, 4s, 8s… up to 60 seconds. Allow 3-5 retry attempts. More than 5 attempts risks offending the user experience when the agent appears frozen. Fewer than 3 attempts gives up too quickly on transient issues that often resolve themselves.
OpenAI recommends retrying 529 errors with exponential backoff. For 429 rate limit errors, parse the retry-after-ms header if present. Use that exact delay instead of your calculated backoff. The provider knows when your quota will reset.
Adding jitter to prevent thundering herd
Jitter randomizes the wait time. Without it, hundreds of agents retrying simultaneously after a provider outage creates a second spike. The tenacity library offers wait_random_exponential which applies decorrelated jitter. Use wait_exponential(multiplier=1, min=1, max=60) + wait_random(0, 1) to add up to 1 second of random delay to each wait period.
You can read more about tuning these parameters in our article covering retry and backoff strategy for ai apis across distributed systems.
When to bail out: circuit breaker integration
Retries alone cannot protect against cascading failures. Wrap your tool execution in a circuit breaker pattern. After a threshold of consecutive failures (e.g., 5), the circuit opens and fast-fails all requests for a reset timeout period (e.g., 30 seconds). This prevents retry storms from overwhelming a struggling service. The circuit breaker pattern comes from Michael Nygard’s “Release It!” and remains essential for production resilience.
What are idempotency keys and how do you generate them?
Idempotency keys are unique tokens generated by the client before executing an operation with side effects. For agent tools, generate keys from (message_id + tool_name + args_hash) for deterministic retries, or use random UUID v4 for non-deterministic executions. Store keys in Redis or a similar persistent store with a TTL exceeding your maximum retry duration to track completion across attempts.
The IETF HTTP API working group maintains a draft specification for the Idempotency-Key header [datatracker.ietf.org]. Services like Stripe and PayPal support this header. When you pass a key, the service returns cached results for duplicate requests instead of re-executing the operation.
Key generation strategies: deterministic vs random
Deterministic keys derive from the request content. For example, hash(message_id + tool_name + json(args, sort_keys=True)). This approach automatically deduplicates identical calls within the same conversation turn. However, it leaks argument patterns into your key storage. Random keys (UUID v4 or ULID) provide privacy but require explicit duplicate detection logic. Use deterministic keys for read-then-write patterns. Use random keys when arguments contain sensitive data.
Here is a simple key generator:
import hashlib
import json
from uuid import uuid4
def generate_idempotency_key(message_id: str, tool_name: str, args: dict, deterministic: bool = True) -> str:
"""Generate an idempotency key for a tool call."""
if deterministic:
args_json = json.dumps(args, sort_keys=True)
payload = f"{message_id}:{tool_name}:{args_json}"
return hashlib.sha256(payload.encode()).hexdigest()
else:
return str(uuid4())Where to store keys: memory, Redis, or tool metadata
In-memory storage (Python dict) works for single-process agents but loses state on restart. Redis persists across restarts and works for distributed agent fleets. Use Redis with SETNX (set if not exists) to atomically claim a key. Store the execution state (pending, completed, failed) and the result payload.
Some tool execution frameworks let you attach metadata to the tool call. LangChain’s tool_call object includes an id field. You can reuse this as your idempotency key if your downstream service supports it.
TTL and cleanup for key storage
Set TTL to exceed your maximum retry duration plus a safety buffer. A 5-minute retry window with 60-second max delays might need 10-15 minutes of TTL. For financial operations, regulatory requirements may mandate 24-72 hour retention. Use Redis EXPIRE or set TTL on write. Implement lazy cleanup by letting keys expire rather than deleting immediately after completion. This handles late-arriving retries from network delays.
The tradeoff is in-memory vs Redis key storage. Memory is faster (microseconds) but loses state on restart. Redis adds milliseconds of latency but survives crashes and enables distributed agent fleets. For production agents, Redis is the right choice.
How to implement agent tool retry in Python
Implement agent tool retry by combining tenacity for exponential backoff with a custom idempotency manager. Wrap tool functions with a decorator that stores execution results keyed by idempotency key in Redis. On duplicate calls, return cached results instead of re-executing. This pattern works with OpenAI, Anthropic, and LangChain frameworks.
Step 1: Setting up tenacity with custom wait strategies
First, install the required packages with pinned versions. Tenacity 8.x or later supports async functions and custom wait strategies.
pip install tenacity==8.5.0 redis==5.2.0 openai==1.52.0Last verified: 2026-10-13
Import tenacity components and define the retry conditions. Retry on connection errors, timeouts, and 5xx status codes. Do not retry on 4xx client errors (except 429).
from tenacity import (
retry,
stop_after_attempt,
wait_exponential_jitter,
retry_if_exception_type,
before_sleep_log,
)
import logging
import asyncio
from httpx import HTTPStatusError, RequestError
logger = logging.getLogger(__name__)
def is_retryable_error(exception):
"""Return True if the exception is safe to retry."""
if isinstance(exception, (RequestError, asyncio.TimeoutError)):
return True
if isinstance(exception, HTTPStatusError):
status = exception.response.status_code
return status >= 500 or status == 429
return False
retry_decorator = retry(
stop=stop_after_attempt(5),
wait=wait_exponential_jitter(multiplier=1, min=1, max=60, jitter=2),
retry=retry_if_exception_type(Exception),
before_sleep=before_sleep_log(logger, logging.WARNING),
retry_error_callback=lambda retry_state: None,
)Step 2: Building the idempotency key manager
Create a class that manages idempotency keys in Redis. It should store pending state, completed results, and handle concurrent access.
import json
import time
from redis import Redis
class IdempotencyManager:
"""Manages idempotency keys and execution state in Redis."""
PENDING_TTL = 300
COMPLETED_TTL = 3600
def __init__(self, redis_client: Redis):
self.redis = redis_client
def start_execution(self, key: str) -> bool:
"""Atomically claim a key. Returns False if already pending or completed."""
stored = self.redis.set(
f"idempotency:{key}:status",
json.dumps({"status": "pending", "timestamp": time.time()}),
nx=True,
ex=self.PENDING_TTL,
)
return stored is not None
def complete_execution(self, key: str, result: dict):
"""Mark execution as completed and store the result."""
self.redis.set(
f"idempotency:{key}:status",
json.dumps({"status": "completed", "result": result, "timestamp": time.time()}),
ex=self.COMPLETED_TTL,
)
def fail_execution(self, key: str, error: str):
"""Mark execution as failed. Allow retry by removing the key."""
self.redis.delete(f"idempotency:{key}:status")
def get_result(self, key: str) -> dict | None:
"""Return cached result if execution already completed."""
data = self.redis.get(f"idempotency:{key}:status")
if data is None:
return None
parsed = json.loads(data)
if parsed.get("status") == "completed":
return parsed.get("result")
return NoneStep 3: Integrating with OpenAI/Anthropic tool calling
Combine the retry decorator with the idempotency manager in a single decorator for agent tools. This example uses synchronous Redis for simplicity.
import functools
from typing import Callable, Any
def agent_tool_retry(
idempotency_manager: IdempotencyManager,
max_attempts: int = 3,
deterministic_keys: bool = True,
):
"""Decorator combining retry logic with idempotency protection."""
def decorator(func: Callable) -> Callable:
@functools.wraps(func)
def wrapper(message_id: str, tool_name: str, args: dict, *call_args, **call_kwargs) -> Any:
key = generate_idempotency_key(message_id, tool_name, args, deterministic_keys)
cached_result = idempotency_manager.get_result(key)
if cached_result is not None:
logger.info(f"Returning cached result for idempotency key {key}")
return cached_result
if not idempotency_manager.start_execution(key):
logger.warning(f"Execution already in progress for key {key}")
raise Exception(f"Duplicate execution attempt for {tool_name}")
attempt = 0
last_error = None
while attempt < max_attempts:
try:
result = func(*call_args, **call_kwargs)
idempotency_manager.complete_execution(key, result)
return result
except Exception as e:
last_error = e
if not is_retryable_error(e):
idempotency_manager.fail_execution(key, str(e))
raise
attempt += 1
wait_time = min(2 ** attempt, 60)
time.sleep(wait_time + (time.time() % 1))
idempotency_manager.fail_execution(key, str(last_error))
raise last_error
return wrapper
return decoratorUse the decorator on your tool functions. Pass the message ID from the LLM conversation to enable deterministic keys.
redis_client = Redis(host="localhost", port=6379, db=0)
idempotency_mgr = IdempotencyManager(redis_client)
@agent_tool_retry(idempotency_manager=idempotency_mgr, max_attempts=4)
def process_refund(message_id: str, payment_id: str, amount_cents: int) -> dict:
"""Process a refund using an external payment API."""
import httpx
response = httpx.post(
"https://api.example.com/refunds",
json={"payment_id": payment_id, "amount": amount_cents},
headers={"Idempotency-Key": message_id},
timeout=30.0,
)
response.raise_for_status()
return response.json()
result = process_refund(
message_id="msg_abc123",
payment_id="pay_xyz789",
amount_cents=1000
)
print(result)Expected output after successful execution:
{"refund_id": "ref_qwerty", "status": "processed", "amount": 1000}On retry with the same message_id and args, the cached result returns immediately without calling the external API.
Step 4: Handling streaming responses with retry
Streaming tool calls fail differently. The connection drops mid-stream, leaving you with incomplete JSON. OpenAI’s Python SDK uses AsyncStream. Anthropic uses Stream. Both emit chunks that you must buffer.
When a stream fails, you cannot simply replay it. The LLM context has moved on. Store the partial result if you received enough to reconstruct the tool call. Otherwise, raise an exception to trigger a full retry. The idempotency key prevents the underlying operation from executing twice even if the stream fails on retry.
For more on building robust agents, see our guide on building a production-ready context-aware ai agent with google adk that handles similar state management challenges.
When should you NOT retry agent tool calls?
Skip retry for 4xx client errors (except 429 rate limits), deterministic failures like validation errors or auth failures, and tools with irreversible non-idempotent side effects where you cannot verify prior execution. Also avoid retry when the LLM context window has advanced, as stale tool results may mislead the agent.
HTTP status code decision tree
Use this table to decide whether to retry based on HTTP status codes:
| Status Code | Category | Retry? | Reason |
|---|---|---|---|
| 400 | Bad Request | No | Client error, fix your code |
| 401 | Unauthorized | No | Invalid API key or expired token |
| 403 | Forbidden | No | Permissions issue, check scopes |
| 404 | Not Found | No | Resource does not exist |
| 422 | Unprocessable | No | Validation failed |
| 429 | Rate Limited | Yes | Respect retry-after header |
| 500 | Server Error | Yes | Transient provider issue |
| 502 | Bad Gateway | Yes | Upstream timeout |
| 503 | Unavailable | Yes | Provider overloaded |
| 504 | Gateway Timeout | Yes | Request took too long |
Irreversible operations: external APIs without idempotency
Some external APIs do not support idempotency keys. Sending an SMS, dispatching a physical package, or triggering an IoT device action may be impossible to undo. For these tools, implement a two-phase pattern. First, insert a record in your database marking the operation as pending. Then, call the external API. If the call fails, you can query the database before retrying to check if the operation already executed. This client-side compensation pattern adds latency but prevents duplicate side effects.
Context staleness in conversation threads
LLM agents maintain conversation context across turns. If a tool call takes 60 seconds to complete with retries, the user may have already sent a new message. The tool result becomes stale. When you detect new user input during a long-running tool call, consider abandoning the retry and letting the agent re-plan its response based on updated context. LangGraph and similar frameworks support interrupting long-running tool calls when new messages arrive.
Production checklist for agent tool resilience
Before deploying agent tool retry with exponential backoff and idempotency to production, verify these critical items to prevent duplicate side effects and ensure reliable retry behavior under load.
- All tools with side effects have idempotency key generation.
- Retry attempts are logged with trace IDs
