RadixAttention prefix caching: vLLM latency reduction

You are watching GPU utilization spike to 100% on every new request despite serving identical system prompts, burning through budget on redundant compute while latency destroys user experience. RadixAttention prefix caching vLLM eliminates this waste by automatically detecting and reusing shared prompt prefixes across requests, reducing time-to-first-token from 800ms to under 100ms in production deployments with high prefix overlap. In this architectural deep-dive, you will learn to architect cache-aware serving strategies, configure eviction policies for your specific workload, and diagnose silent caching failures that standard metrics miss.

What is RadixAttention and why does vLLM need it?

RadixAttention is vLLM’s radix tree-based key-value cache indexing system that enables automatic prefix sharing across requests, eliminating redundant prefill computation by reusing cached blocks from memory-mapped prompt prefixes. Itextends vLLM’s existing PagedAttention by adding a structural index over cached token sequences, allowing the engine to detect and exploit prefix overlap without any client-side modifications [arxiv.org].

The prefill bottleneck in transformer inference

Every LLM request begins with a prefill phase where the model processes the entire input prompt to populate the key-value (KV) cache. This phase is compute-intensive and scales with prompt length. For a 1000-token prompt on a 70B parameter model, prefill can consume hundreds of milliseconds of GPU time. When multiple requests share identical prefixes (system prompts, few-shot examples, retrieved context), standard attention implementations recompute the entire prefix for each request. This redundancy wastes GPU cycles and inflates time-to-first-token (TTFT), directly impacting user-perceived latency.

From PagedAttention to RadixAttention: the evolution

vLLM initially solved memory fragmentation through PagedAttention, which manages the KV cache as fixed-size blocks similar to operating system virtual memory. PagedAttention uses reference counting to handle shared blocks within a single sequence (like during beam search). RadixAttention extends this concept. It builds a radix tree index over these blocks, enabling automatic prefix matching and reuse across completely unrelated requests from different users. While PagedAttention asks “which blocks belong to this sequence?”, RadixAttention asks “have I seen this token prefix before, and can I reuse it?” [github.com].

How radix trees map prefixes to physical GPU blocks

A radix tree is a space-optimized trie where each node represents a sequence of tokens rather than a single token. In vLLM’s implementation, nodes correspond to one or more physical blocks in the KV cache. When a new prompt arrives, vLLM walks the radix tree from the root, matching tokens until it cannot continue. The deepest matched node represents the longest cached prefix. The engine allocates new blocks only for the unmatched suffix, linking them to the existing cached prefix.

How does RadixAttention prefix caching actually work?

RadixAttention builds a radix tree over token sequences where each node tracks cached KV blocks. Incoming prompts traverse this tree to find matching prefixes, enabling block-level reuse without copying tensors. The block manager maintains reference counts to prevent premature eviction while divergent suffixes branch off as new tree nodes [arxiv.org].

Token-to-block mapping in the radix tree

vLLM’s BlockManagerV2 divides the KV cache into fixed-size blocks (typically 16 or 32 tokens per block). The radix tree operates at block granularity. Each tree node stores:

• A token sequence (the prefix leading to this node)
• Block IDs pointing to physical GPU memory
• A reference count for safe eviction

When prompt A (“System: You are a helpful assistant. User: Hello”) and prompt B (“System: You are a helpful assistant. User: What is Python?”) arrive, they share the system prompt prefix. The radix tree creates a shared node for “System: You are a helpful assistant.” Both requests reference the same physical blocks. Only the diverging suffixes (“Hello” vs “What is Python?”) consume new memory.

Physical vs logical block management

The separation of logical and physical blocks is crucial. Logical blocks represent what the model “sees” as a contiguous sequence. Physical blocks are the actual memory locations in GPU VRAM. RadixAttention maintains this indirection. Multiple logical sequences can map to the same physical blocks when they share a prefix. This mapping is transparent to the attention kernel, which operates on logical block indices while the block manager handles the physical address translation.

The matching algorithm: longest prefix wins

The core matching routine in PrefixCacher computes a hash for incoming token sequences and walks the tree:

1. Hash the token sequence at block boundaries.
2. Probe the radix tree starting from the root.
3. Compare stored tokens at each node to verify matches (handling hash collisions).
4. Stop at the deepest matching node.
5. Return the list of physical block IDs for the matched prefix.

If the prompt is “System prompt X” and the tree contains “System prompt X” + “User question Y”, the algorithm matches the full “System prompt X” prefix. The new request reuses those blocks entirely.

Copy-on-write semantics for diverging suffixes

When two requests share a prefix but diverge later, RadixAttention employs copy-on-write (COW) semantics. Initially, both share the exact same physical blocks. As generation proceeds, if one sequence modifies KV entries (which generally does not happen for prefix tokens in autoregressive models), or more commonly, when eviction pressure forces reallocation, the reference counts manage ownership. More importantly, when the shared prefix is actively being used by ongoing generation, the block manager increments reference counts. Eviction policies respect these counts, ensuring active prefixes remain in memory.

When should you enable prefix caching in production?

Enable prefix caching when your workload has high prefix similarity, sufficient GPU memory for tree overhead, and strict TTFT requirements. You should measure your actual prefix overlap ratio; RadixAttention delivers value only when this ratio exceeds your memory overhead threshold, typically benefiting chat templates, multi-turn conversations, and document Q&A systems most significantly [docs.vllm.ai].

Workload signal analysis: measuring prefix overlap

Before enabling caching, analyze your traffic. Tools like vLLM’s built-in metrics expose prefix_cache_hit_rate, but you should estimate this before deployment. Consider these workload types with their typical hit rate potential:

Workload Type	Prefix Overlap	Expected Hit Rate	Caching Benefit
Chat applications with fixed system prompts	30-50% identical	70-90%	High – major TTFT gains
Multi-turn conversations	60-80% cumulative	80-95%	Very high – compounding savings
RAG with retrieved context	10-30% template overlap	20-50%	Moderate – depends on template
Diverse prompts without templates	<5% overlap	<10%	Low – overhead may exceed benefit

For chat applications, if 80% of requests use the same 500-token system prompt, you effectively save 400 tokens of prefill per request. At 1000 requests per minute, this compounds rapidly.

Memory budget calculations: tree overhead + block pool

RadixAttention adds memory overhead for two components:

1. The radix tree node structures (metadata)
2. The cached KV blocks themselves

The radix tree typically adds 2-5% memory overhead for metadata, plus 1-3% for tree node structures, scaling with unique prefix count rather than sequence length [arxiv.org]. To calculate your budget:

# Estimate RadixAttention memory requirements
vram_total_gb = 80  # A100 80GB
model_weights_gb = 35  # Llama-3-70B in FP16
kv_cache_fraction = 0.30  # Typical allocation
available_for_kv_gb = vram_total_gb - model_weights_gb
kv_cache_budget_gb = available_for_kv_gb * kv_cache_fraction

# Assuming FP16, 2048 context, block_size=16
tokens_per_block = 16
bytes_per_token = 2  # FP16
layer_memory_per_token = bytes_per_token * 2  # K and V
total_layers = 80  # For 70B model
memory_per_token = layer_memory_per_token * total_layers

# Calculate block capacity
block_size_bytes = tokens_per_block * memory_per_token
block_capacity = int((kv_cache_budget_gb * 1024**3) / block_size_bytes)

print(f"Available blocks: {block_capacity:,}")
print(f"Estimated prefix cache overhead: {block_capacity * 0.05:,.0f} blocks")

Output:

Available blocks: 245,760
Estimated prefix cache overhead: 12,288 blocks

This overhead reduces your maximum batch size. If your workload cannot tolerate a 5-10% reduction in batch capacity, think carefully about enabling caching.

Anti-patterns: when caching hurts more than helps

Not every workload benefits. Avoid enabling prefix caching in these scenarios:

• Highly diverse prompts: If users submit unique prompts with no shared structure, the radix tree becomes a burden. Lookups find no matches, but still consume CPU cycles walking the tree.
• Memory-constrained deployments: On GPUs with tight memory budgets (like T4 or L4), overhead reduces batch size, potentially lowering throughput more than cache hits improve it.
• Single-request batch processing: Offline batch jobs with unique documents benefit minimally. Each document is processed once; there is no repetition to exploit.

How to configure vLLM RadixAttention for maximum efficiency

Configure RadixAttention with --enable-prefix-caching, tune block allocation, set appropriate batch token limits, and monitor hit rate metrics. You should also consider block size selection and eviction policy tuning based on your specific workload pattern.

CLI flags and engine arguments

The primary configuration is straightforward:

python -m vllm.entrypoints.openai.api_server \
    --model meta-llama/Meta-Llama-3.1-70B-Instruct \
    --enable-prefix-caching \
    --gpu-memory-utilization 0.90 \
    --max-model-len 8192 \
    --max-num-batched-tokens 32768 \
    --block-size 16

Key flags explained:

• --enable-prefix-caching: Activates RadixAttention. Without this, vLLM uses standard PagedAttention without cross-request caching.
• --block-size: Controls token granularity. Smaller blocks (16) offer better prefix matching precision but increase tree management overhead. Larger blocks (64, 128) reduce overhead but may waste memory on partial matches.
• --max-num-batched-tokens: Determines the maximum prefill chunk size. Smaller chunks interact better with incremental prefix matching.

You can also configure programmatically:

from vllm import LLM, SamplingParams
from vllm.config import CacheConfig

llm = LLM(
    model="meta-llama/Meta-Llama-3.1-70B-Instruct",
    enable_prefix_caching=True,
    gpu_memory_utilization=0.90,
    max_model_len=8192,
    block_size=16,
)

Block size selection: 16 vs 32 vs 128

Block size is a critical tuning parameter. The default of 16 tokens balances precision and overhead for most workloads.

Block Size	Tree Overhead	Match Precision	Best For
16	Higher	Fine-grained (16 tokens)	Chat with varied prompts
32	Lower	Coarser (32 tokens)	Long documents, RAG
64	Low	Coarse	Batch processing with large prefixes
128	Very low	Very coarse	All-or-nothing template caching

If your system prompts are exactly 512 tokens, a block size of 16 captures matches precisely. A block size of 64 might cut through the middle of your template, preventing full reuse.

Eviction policy tuning: LRU vs frequency-based

vLLM currently uses a least-recently-used (LRU) eviction policy for both regular KV cache blocks and prefix cache entries. The policy evicts the oldest unused prefix nodes first. You cannot switch eviction policies at runtime as of vLLM 0.6.x. For chat workloads with temporal locality (users within a session share system prompts), LRU performs well. For batch processing where certain common prefixes (like a fixed template) should remain cached regardless of recency, you may need to implement custom warmup routines to prime the cache on startup.

Dynamic memory allocation with chunked prefill

Chunked prefill, enabled by default in recent vLLM versions, breaks large prompts into smaller chunks. This interacts with RadixAttention. If a prefix spans multiple prefill chunks, the block manager must ensure the radix tree nodes are correctly linked as each chunk completes. For most workloads, keep chunked prefill enabled as it improves throughput by interleaving prefill and decode operations. Monitor prefix_cache_hit_rate when enabling chunked prefill to ensure it does not interfere with prefix matching during incremental processing.

Production debugging: when prefix caching fails silently

Silent failures include hash collisions on similar-but-different prefixes, block fragmentation causing artificial misses, and tokenizer mismatches. You need to enable detailed block manager logging and verify cache hits through metrics rather than assuming caching works automatically.

Verifying cache hits with vLLM logs

vLLM exposes a Prometheus metrics endpoint. Query the hit rate directly:

curl http://localhost:8000/metrics | grep prefix_cache

Expected output:

vllm:prefix_cache_hit_rate 0.847
vllm:prefix_cache_queries_total 15234
vllm:prefix_cache_hits_total 12913

A hit rate near 0.0 suggests caching is enabled but not functioning. Common causes:

1. Cache is warming up (first requests always miss).
2. Tokenization differs between requests.
3. Prefix exceed available memory, causing immediate eviction.

Enable debug logging to trace block allocation:

import logging
logging.basicConfig(level=logging.DEBUG)
logging.getLogger("vllm.core.block_manager").setLevel(logging.DEBUG)

Look for lines like “Allocated X blocks for prefix hash Y” and “Cache hit for prefix: [token_ids]”.

Memory fragmentation and block allocator exhaustion

High prefix diversity causes fragmentation. If many unique prefixes enter the cache, the radix tree grows, consuming metadata memory. Physical blocks pin to specific prefixes. Eventually, the block allocator cannot find contiguous space for new suffixes even if total free memory appears sufficient. This manifests as successful prefix matches followed by allocation failures for the suffix.

Monitor vllm:num_gpu_blocks_allocated and vllm:num_requests_waiting. If waiting requests increase while allocated blocks plateau, you may have hit fragmentation limits. The fix involves tuning --garbage-collector frequency or reducing --block-size for more granular allocation.

Tokenizer normalization gotchas

Why is vLLM not using my prefix cache even with --enable-prefix-caching? Common causes include tokenization mismatches. Spaces, special tokens, and formatting differences invalidate cache hits. The prompt “Hello world” and “Hello world” (double space) hash to different cache keys. The model sees them differently, so the cache correctly distinguishes them, but you expect a hit that never comes.

Check tokenization:

from transformers import AutoTokenizer
tok1 = AutoTokenizer.from_pretrained("meta-llama/Meta-Llama-3.1-70B-Instruct")

print(tok1.encode("Hello world"))
print(tok1.encode("Hello  world"))
# Different outputs means different cache keys

Normalize prompts on the client side. Strip extra spaces, canonicalize JSON structures, and ensure consistent special token usage across requests targeting the same prefix.

The false hit bug: duplicate token sequences with different meanings

A rare edge case involves different prompts that tokenize identically. If “Hello world” from a casual chat and “Hello world” from a formal instruction tokenize the same way, they will share the cache. This is usually correct behavior. The prefix is identical. However, if you have prompt templates where identical early tokens have different continuation contexts, you may observe unexpected behavior. vLLM treats this as a feature, not a bug. The prefix computation is genuinely identical. The divergence occurs at the suffix, which naturally branches in the radix tree.

Comparing RadixAttention to alternative prefix caching approaches

RadixAttention provides automatic client-transparent caching versus manual prompt templating or external cache stores. It trades fine-grained control for zero integration overhead, offering superior usability compared to explicit cache management systems.

Approach	Integration Effort	Flexibility	Performance	Best For
vLLM RadixAttention	Zero (single flag)	Automatic detection	High (block-level)	Production serving
TensorRT-LLM Prompt Tables	Medium (template registration)	Explicit control	High (optimized lookups)	Batch inference
External KV Cache (Redis)	High (custom logic)	Full control	Medium (network overhead)	Multi-node setups
Hugging Face TGI	Low (config flag)	Automatic	Medium	Single-GPU deployments

TensorRT-LLM requires explicit prompt table registration. You must identify and register templates in advance. This provides control but fails on unexpected prompt patterns. RadixAttention automatically adapts to any pattern, including prompts you did not anticipate during development.

External KV cache stores like Redis caching patterns: 5 ways to slash latency offer flexibility for distributed architectures. However, moving KV tensors across the network introduces latency. Intra-GPU block sharing in RadixAttention operates at memory bandwidth speeds, orders of magnitude faster than network transfers.

Hugging Face TGI supports prefix caching but with limitations on multi-GPU setups and without the sophisticated eviction policies of vLLM’s radix tree implementation.

RadixAttention performance: benchmarks and trade-offs

RadixAttention achieves 5-10x TTFT reduction on prefix-heavy workloads but adds 5-15% memory overhead. It can degrade throughput on low-similarity workloads due to lookup overhead. Understanding these trade-offs helps you decide when to enable caching.

Measuring TTFT vs throughput trade-offs

Benchmark your specific workload. Use vLLM’s benchmarking scripts:

git clone https://github.com/vllm-project/vllm.git
cd vllm/benchmarks
python benchmark_serving.py \
    --model meta-llama/Meta-Llama-3.1-70B-Instruct \
    --backend vllm \
    --dataset-prefix "System: You are a helpful assistant." \
    --request-rate 10 \
    --num-prompts 1000

Compare TTFT with and without prefix caching:

# Without prefix caching
python benchmark_serving.py --model ... --max-tokens 100

# With prefix caching
python benchmark_serving.py --model ... --max-tokens 100 --enable-prefix-caching

Typical results for a chat workload with 500-token shared system prompt:

Metric	Without Caching	With Caching	Improvement
Mean TTFT	824ms	98ms	88% reduction
P99 TTFT	1120ms