Redis rate limiting for AI gateway resilience

Your LLM gateway just cratered under a traffic spike. OpenAI rate limits hit, fallback logic never triggered, and Redis became the bottleneck instead of the savior. A properly architected AI gateway rate limiting Redis implementation handles 50,000 req/sec with sub-5ms p99 latency on Redis Cluster 7.2. This guide shows you how to build a gateway that degrades gracefully when primary APIs fail, never overruns rate limits, and exposes health metrics for observability.

What makes an AI gateway resilient vs just distributed?

Resilient AI gateways combine distributed rate limiting, health-aware routing, and failure-isolated degradation. Redis enables atomic counters and fast state for both functions, but only with proper Lua scripting and cluster topology awareness. A distributed gateway spreads load. A resilient gateway survives partial failures without cascading outages.

Latency budgets: why 10ms matters at 10k rps

Every millisecond added to your gateway layer compounds. At 10,000 requests per second, a 10ms latency budget means you have 100 concurrent requests in flight just from gateway overhead. Redis Cluster 7.2 achieves sub-millisecond response times for simple key operations, but atomic rate limiting scripts with multiple keys push toward 3-5ms. Your budget must account for network round trips, script execution, and the actual model API call.

Failure domains: gateway vs model provider vs Redis

You have three distinct failure domains. The gateway itself can crash or run out of file descriptors. Model providers like OpenAI or Anthropic return 429 rate limit errors, 503 service unavailable, or experience regional outages. Redis can lose connectivity, hit memory limits, or suffer cluster rebalancing delays. Each domain requires independent health tracking and fallback strategies. A failure in Redis should not take down the entire gateway. You can see similar resilience patterns in our zero-trust api gateway design: 5 patterns article.

The 3-layer resilience model

Layer one is the application-level circuit breaker for each model provider. Layer two is the Redis-backed distributed rate limiter acting as a global coordination point. Layer three is local in-memory caching for fail-open scenarios when Redis becomes unavailable. This model prevents a single Redis outage from causing a complete service outage.

How does Redis rate limiting work for token-based AI APIs?

Redis stores sliding window counters for requests and token budgets separately. Lua scripts ensure atomic decrement-and-check operations across cluster nodes, preventing race conditions that cause limit overruns. The system tracks both request counts and token consumption because AI APIs enforce separate limits for each dimension.

Fixed vs sliding window vs token bucket for LLMs

Fixed windows cause issues at boundary edges. A user sending 99 requests at 11:59 and 99 more at 12:01 could double their quota in a two-minute span. Sliding windows fix this but require storing more data points. Token bucket algorithms handle burst-then-quiescent LLM workloads better than fixed windows. Use separate buckets for requests-per-minute and tokens-per-minute. Size the tokens-per-minute bucket to your tier’s limit plus 10% headroom for streaming overhead. According to [platform.openai.com], token limits vary significantly by tier, and your rate limiter must track both dimensions independently.

Lua script for atomic multi-key rate check

Redis guarantees Lua script execution is atomic. No other command runs while the script executes. This prevents the race condition where two requests read the same counter value, both see quota available, and both decrement simultaneously, causing an overrun. The script below checks both request and token limits in a single atomic operation:

-- rate_limiter.lua
-- KEYS[1] = request counter key, KEYS[2] = token counter key
-- ARGV[1] = window start timestamp, ARGV[2] = window duration in seconds
-- ARGV[3] = request limit, ARGV[4] = token cost, ARGV[5] = token limit
-- Returns: {allowed: 0/1, remaining_requests, remaining_tokens, reset_time}

local request_key = KEYS[1]
local token_key = KEYS[2]
local window_start = tonumber(ARGV[1])
local window_duration = tonumber(ARGV[2])
local request_limit = tonumber(ARGV[3])
local token_cost = tonumber(ARGV[4])
local token_limit = tonumber(ARGV[5])

-- Clean expired entries from sorted sets
redis.call('ZREMRANGEBYSCORE', request_key, 0, window_start - window_duration * 1000)
redis.call('ZREMRANGEBYSCORE', token_key, 0, window_start - window_duration * 1000)

local current_requests = redis.call('ZCARD', request_key)
local current_tokens = redis.call('ZCARD', token_key)

if current_requests >= request_limit then
    return {0, 0, 0, window_start + window_duration * 1000}
end

if current_tokens + token_cost > token_limit then
    return {0, 0, token_limit - current_tokens, window_start + window_duration * 1000}
end

-- Add entries for this request
local request_id = tostring(window_start) .. ':' .. tostring(math.random())
redis.call('ZADD', request_key, window_start, request_id)
redis.call('ZADD', token_key, window_start, request_id .. ':tokens:' .. tostring(token_cost))

-- Set expiry
redis.call('EXPIRE', request_key, window_duration)
redis.call('EXPIRE', token_key, window_duration)

return {1, request_limit - current_requests - 1, token_limit - current_tokens - token_cost, window_start + window_duration * 1000}

This script uses sorted sets to track individual requests within the sliding window. The ZREMRANGEBYSCORE command removes expired entries atomically. Each request adds a unique member to the sorted set with its timestamp as the score.

Handling burst vs sustained load patterns

LLM workloads exhibit burst patterns. A user sends ten prompts rapidly, then goes quiet for minutes. Your rate limiter must allow reasonable bursts while preventing sustained abuse. Configure the token bucket refill rate to match the average expected load, but set the bucket capacity to allow for short bursts. For example, if a user has a 1000 tokens-per-minute limit, set a bucket capacity of 500 tokens that refills at 16 tokens per second.

What is fallback routing and when should it trigger?

Fallback routing redirects traffic to backup model providers when primary APIs fail health checks or exhaust rate limits. Effective implementations use weighted scoring based on latency, error rate, and remaining quota. This creates a Redis distributed rate limiter AI API architecture that survives provider outages.

Health check design: passive vs active probes

Passive health checks monitor actual API responses. A 429 status code indicates rate limit exhaustion. A 503 indicates temporary unavailability. Track these responses in Redis with expiry counters to build a health score without sending additional traffic. Active probes send periodic synthetic requests to verify provider availability. Passive checks are cheaper but may miss outages until real traffic hits them. Active probes add latency and cost but detect issues earlier. Use passive monitoring for high-traffic routes and active probes for low-traffic endpoints.

Weighted round-robin with penalty decay

Static priority lists fail when a high-priority provider degrades. Instead, score providers dynamically using the formula: score = (success_rate × 0.4) + (1/latency_p99 × 0.3) + (remaining_quota_ratio × 0.3). Decay scores exponentially with a 60-second half-life. Route to the highest-scoring provider, excluding those below a health threshold. This approach adapts to changing provider conditions without manual intervention.

Cascading failure prevention: circuit breaker integration

A circuit breaker prevents repeated calls to a failing service. When a provider returns three consecutive errors within a 10-second window, open the circuit and fail fast for the next 30 seconds. After the timeout, allow a single request through. If it succeeds, close the circuit. If it fails, extend the timeout. This fallback routing AI gateway architecture prevents thundering herds from overwhelming a recovering provider. According to [martinfowler.com], the circuit breaker pattern provides this exact failure isolation.

How do you implement distributed rate limiting with Redis Cluster?

Redis Cluster 7.x uses hash tags for co-located counters, Lua scripts for atomic operations, and replica reads for non-critical checks. Handle ASK/MOVED redirects and cluster rebalancing gracefully to maintain availability.

Hash tags for rate limit key locality

Redis Cluster distributes keys across slots based on the key name. Multi-key operations require all keys to reside on the same slot. Hash tags force key locality by wrapping a common portion of the key in curly braces. Keys like {user_123}:requests and {user_123}:tokens are guaranteed to land on the same cluster node. Without hash tags, your Lua script would fail with a CROSSSLOT error when trying to operate on multiple keys.

Handling CLUSTERDOWN scenarios

During cluster rebalancing or master failures, Redis may return CLUSTERDOWN errors. Your gateway must fail open with cached quotas or queue requests temporarily. Implement client-side timeouts for Redis operations. If Redis does not respond within 100ms, proceed with the last known quota state and log the incident for investigation. We discussed similar failure scenarios in our designing a high-concurrency flash sale stock & inventory reservation system with node.js, redis, lua and mongodb article.

Read replicas for quota checks, primaries for decrements

Read replicas reduce load on primaries and provide lower latency for quota checks. Use replicas for read-only operations like checking remaining quota. Always use primaries for decrement operations to ensure strong consistency. According to [redis.io], Redis Cluster allows read-from-replica configuration, but you must handle potential staleness. A brief delay in quota visibility is acceptable for user experience, but never allow overruns due to stale reads.

What are the production failure modes you’ve actually hit?

Redis Cluster split-brain causes double-counting when nodes accept writes independently. Lua script timeouts occur during hot-key contention. Fallback storms happen when all providers hit limits simultaneously. These failure modes require explicit handling in a Redis circuit breaker LLM gateway.

Hot key amplification: mitigation strategies

A single user making thousands of requests per second creates a hot key on one cluster node. This node becomes a bottleneck even if the rest of the cluster is idle. Mitigate by sharding user keys across multiple slots using a hash of the user ID modulo a configurable shard count. For example, {user_123}:requests:shard_5 spreads the load across different nodes. Alternatively, use multiple Redis clusters for different user segments.

The thundering fallback anti-pattern

When a primary provider fails, all gateways in your fleet simultaneously detect the failure and switch to fallback providers. This swamps the fallback providers with a sudden traffic surge, causing them to fail as well. Prevent this by implementing randomized backoff times before switching providers. Stagger health check intervals across gateway instances. Use a distributed coordination mechanism like Redis distributed locks to elect a single gateway to perform active health probes.

When Redis itself becomes the bottleneck

Redis can become the single point of failure if all rate limit checks must synchronously wait for it. Implement local in-memory caching for rate limit decisions with a short TTL, such as 100ms. This allows the gateway to serve requests from cache during brief Redis outages. Use Lua script timeouts set to 100ms via the lua-time-limit configuration. If a script times out, Redis enters a blocking state where it only accepts SCRIPT KILL or SHUTDOWN NOSAVE. Monitor script execution times and alert before they become critical.

Complete implementation: Python async gateway with Redis backend

This implementation demonstrates a multi-tenant rate limiting OpenAI API gateway with fallback routing. The architecture uses FastAPI for HTTP handling, aio-redis for async Redis connections, and custom middleware for rate limiting logic.

Architecture diagram and dependency graph

The gateway consists of four components. The HTTP handler receives requests and extracts tenant IDs from API keys. The rate limiter checks quotas against Redis. The fallback router selects the appropriate upstream provider. The circuit breaker tracks provider health.

flowchart LR
    A[Client Request] --> B[FastAPI Handler]
    B --> C[Tenant Resolver]
    C --> D[Rate Limiter]
    D --> E{Quota Available?}
    E -- Yes --> F[Fallback Router]
    E -- No --> G[Return 429]
    F --> H[Circuit Breaker]
    H --> I[Provider A]
    H --> J[Provider B]
    H --> K[Provider C]
    I --> L[Response]
    J --> L
    K --> L
    L --> M[Update Metrics]
    M --> N[Update Quota]
    N --> O[Return Response]

Rate limiter class with Redis Lua integration

The following Python code implements the core rate limiting logic. It requires Python 3.10+, FastAPI, and redis-py.

# rate_limiter.py
import time
import hashlib
import asyncio
from typing import Optional, Tuple
from dataclasses import dataclass
import redis.asyncio as redis
from redis.asyncio import Redis
from redis.asyncio.cluster import RedisCluster

@dataclass
class RateLimitResult:
    allowed: bool
    remaining_requests: int
    remaining_tokens: int
    reset_after_ms: int
    retry_after_ms: Optional[int] = None

class DistributedRateLimiter:
    """
    Redis-backed distributed rate limiter with sliding window algorithm.
    Implements multi-tenant rate limiting OpenAI API quotas.
    """
    
    LUA_SCRIPT = """
    local request_key = KEYS[1]
    local token_key = KEYS[2]
    local now = tonumber(ARGV[1])
    local window_ms = tonumber(ARGV[2])
    local request_limit = tonumber(ARGV[3])
    local token_cost = tonumber(ARGV[4])
    local token_limit = tonumber(ARGV[5])
    
    local window_start = now - window_ms
    
    redis.call('ZREMRANGEBYSCORE', request_key, 0, window_start)
    redis.call('ZREMRANGEBYSCORE', token_key, 0, window_start)
    
    local current_requests = redis.call('ZCARD', request_key)
    local request_entries = redis.call('ZRANGE', token_key, 0, -1, 'WITHSCORES')
    local current_tokens = 0
    for i = 1, #request_entries, 2 do
        current_tokens = current_tokens + tonumber(string.match(request_entries[i], ':(%d+)$') or '0')
    end
    
    if current_requests >= request_limit then
        return {0, 0, 0, now + window_ms, window_ms}
    end
    
    if current_tokens + token_cost > token_limit then
        return {0, current_requests, token_limit - current_tokens, now + window_ms, window_ms}
    end
    
    local req_id = now .. ':' .. math.random()
    local token_id = req_id .. ':tokens:' .. token_cost
    
    redis.call('ZADD', request_key, now, req_id)
    redis.call('ZADD', token_key, now, token_id)
    redis.call('EXPIRE', request_key, math.ceil(window_ms / 1000))
    redis.call('EXPIRE', token_key, math.ceil(window_ms / 1000))
    
    return {1, request_limit - current_requests - 1, token_limit - current_tokens - token_cost, now + window_ms, 0}
    """
    
    def __init__(self, redis_url: str, cluster_mode: bool = True):
        self.redis_url = redis_url
        self.cluster_mode = cluster_mode
        self._redis: Optional[RedisCluster | Redis] = None
        self._script_sha: Optional[str] = None
    
    async def connect(self) -> None:
        """Initialize Redis connection."""
        if self.cluster_mode:
            self._redis = await RedisCluster.from_url(self.redis_url)
        else:
            self._redis = await redis.from_url(self.redis_url)
        
        # Register Lua script
        self._script_sha = await self._redis.script_load(self.LUA_SCRIPT)
    
    def _get_keys(self, tenant_id: str) -> Tuple[str, str]:
        """Generate hash-tagged keys for tenant."""
        # Hash tags ensure keys land on same cluster node
        tag = f"{{{tenant_id}}}"
        return f"{tag}:requests", f"{tag}:tokens"
    
    async def check_rate_limit(
        self,
        tenant_id: str,
        token_cost: int,
        request_limit: int,
        token_limit: int,
        window_ms: int = 60000
    ) -> RateLimitResult:
        """
        Check if request is allowed under rate limits.
        
        Args:
            tenant_id: Unique identifier for the tenant
            token_cost: Number of tokens this request will consume
            request_limit: Maximum requests per window
            token_limit: Maximum tokens per window
            window_ms: Window duration in milliseconds
            
        Returns:
            RateLimitResult with allowance status and remaining quota
        """
        if not self._redis:
            raise RuntimeError("Redis not connected. Call connect() first.")
        
        request_key, token_key = self._get_keys(tenant_id)
        now = int(time.time() * 1000)
        
        try:
            result = await self._redis.evalsha(
                self._script_sha,
                2,
                request_key,
                token_key,
                str(now),
                str(window_ms),
                str(request_limit),
                str(token_cost),
                str(token_limit)
            )
            
            return RateLimitResult(
                allowed=bool(result[0]),
                remaining_requests=result[1],
                remaining_tokens=result[2],
                reset_after_ms=result[3],
                retry_after_ms=result[4] if result[0] == 0 else None
            )
        except redis.exceptions.NoScriptError:
            # Script expired, re-register and retry
            self._script_sha = await self._redis.script_load(self.LUA_SCRIPT)
            return await self.check_rate_limit(
                tenant_id, token_cost, request_limit, token_limit, window_ms
            )
        except redis.exceptions.RedisError as e:
            # Fallback: allow request with warning logging
            # In production, integrate with your logging/monitoring system
            print(f"Redis error: {e}. Failing open.")
            return RateLimitResult(
                allowed=True,
                remaining_requests=0,
                remaining_tokens=0,
                reset_after_ms=now + window_ms
            )

This implementation handles the core rate limiting logic with proper error handling. The hash tag syntax {tenant_id} ensures both requests and token counters for a tenant reside on the same Redis Cluster node.

Fallback router with health scoring

The fallback router implements cost-aware routing that weights providers by reliability and performance metrics.

# fallback_router.py
import time
import asyncio
from typing import Dict, List, Optional
from dataclasses import dataclass, field
from enum import Enum
import httpx