KV cache offloading NVMe: vLLM vs TensorRT-LLM

You’re hitting OOM at 32K context despite an A100 80GB, and the “just buy more GPUs” answer is not in the budget. TensorRT-LLM’s KV cache manager achieves roughly 2.3 GB/s read bandwidth from Gen4 NVMe, while vLLM’s CPU offloader peaks at around 800 MB/s under contention. This article provides a production-grade architectural comparison of KV cache offloading NVMe vLLM TensorRT-LLM so you can select the correct offloading architecture for your latency and throughput constraints, deploy with confidence, and know exactly which failure modes to monitor.

What is KV cache offloading and when do you need it?

KV cache offloading moves key-value tensors from GPU HBM to CPU DRAM or NVMe SSD when context exceeds available VRAM. This mechanism is required when serving long-context models on memory-constrained hardware, enabling context lengths far beyond native GPU capacity at the cost of increased latency for cache misses.

The memory wall: why 80GB is not enough for 128K context

A 70B parameter model at FP8 requires roughly 70GB for weights alone. At 128K context, the KV cache grows to approximately 40-60GB. An 80GB A100 cannot hold both without offloading or aggressive quantization. The KV cache scales linearly with sequence length, number of layers, and hidden dimension, making memory the primary bottleneck for long-context inference.

Offloading tiers: CPU DRAM vs NVMe SSD latency hierarchy

CPU DRAM offers 50-100ns latency with 100-200 GB/s bandwidth on DDR5 systems. NVMe SSDs add 10-50 microseconds of latency but provide persistent storage with 3-14 GB/s throughput. GPU HBM sits at sub-microsecond latency with 2-3 TB/s bandwidth. Offloading to CPU DRAM is faster but capacity-limited, while NVMe extends capacity further at a latency cost.

When offloading beats quantization or sparsity

Offloading preserves model accuracy by keeping weights in higher precision while storing only the KV cache externally. Quantization reduces memory for both weights and KV cache, potentially degrading output quality. Choose offloading when accuracy is non-negotiable, context length varies dramatically, or you need to maximize throughput on existing hardware without retraining or calibration.

vLLM KV cache offloading: CPU offloader and v1 attention backend

vLLM implements KV cache offloading through the CPU offloader with block-sparse attention in the v1 attention backend, using prefix caching to reduce redundant computation across requests. As of vLLM 0.6.x, native NVMe offloading is not available in the stable release; the CPU offloader spills to CPU DRAM only [docs.vllm.ai].

Architecture: block manager v2 and external memory tiers

vLLM’s PagedAttention algorithm divides the KV cache into fixed-size blocks managed by a block manager. The CPU offloader extends this by allowing blocks to reside in CPU DRAM when GPU memory fills. The scheduler swaps blocks between GPU and CPU transparently during prefill and decode. The v1 attention backend [docs.vllm.ai] provides optimized kernels for this tiered memory access.

Configuration: enable_prefix_caching, cpu_offload_gb, block_size tuning

Configure CPU offloading by setting the cpu_offload_gb argument to reserve CPU memory for the KV cache. Enable enable_prefix_caching to reuse KV blocks across requests with shared prompts. The block_size parameter (default 16 tokens) affects memory efficiency: smaller blocks reduce fragmentation but increase metadata overhead.

from vllm import LLM, SamplingParams

llm = LLM(
    model="meta-llama/Llama-3.1-70B",
    tensor_parallel_size=2,
    gpu_memory_utilization=0.90,
    swap_space=4,  # CPU swap space in GB
    enforce_eager=True,
)

sampling_params = SamplingParams(
    max_tokens=512,
    temperature=0.7,
)

Limitations: no native NVMe tier, CPU RAM as single spillover

The current vLLM CPU offloader treats CPU DRAM as the only spillover tier. There is no native NVMe offloading path in stable releases. CPU RAM capacity becomes the hard limit, typically 256-512GB on commodity servers. For contexts exceeding CPU DRAM capacity, you must use model parallelism or switch to an engine with NVMe support.

TensorRT-LLM KV cache management: native NVMe and multi-block-mode

TensorRT-LLM provides native KV cache offloading to NVMe via the KV cache manager with multi-block-mode, supporting heterogeneous memory hierarchies and explicit prefetching for latency hiding. TensorRT-LLM typically achieves lower latency for long contexts when NVMe offloading is active due to optimized kernels and explicit memory tier management [nvidia.github.io].

Architecture: KVCacheManager, MemoryPool, and BlockManager abstractions

TensorRT-LLM separates KV cache management into three components: KVCacheManager handles allocation and lifecycle, MemoryPool abstracts GPU/CPU/NVMe tiers, and BlockManager tracks block ownership across tiers. The kv_cache_config JSON specifies tier ratios and eviction policies. This architecture enables fine-grained control over data placement [github.com/NVIDIA/TensorRT-LLM].

Configuration: kv_cache_config, enable_block_reuse, multi_block_mode

The kv_cache_config JSON file defines the offloading behavior. Set enable_block_reuse to true for KV cache reuse across requests. Enable multi_block_mode to allow concurrent block allocation across multiple attention heads. The offload_dir specifies the NVMe mount point.

{
  "KVCacheManager": {
    "enable_block_reuse": true,
    "max_blocks_per_sequence": 128,
    "tokenizer": "meta-llama/Llama-3.1-70B"
  },
  "MemoryPool": {
    "gpu_memory_fraction": 0.85,
    "cpu_memory_fraction": 0.5,
    "offload_dir": "/mnt/nvme-kv-cache",
    "enable_nvme_offload": true,
    "io_parallelism": 4
  }
}

Build the engine with the configuration:

trtllm-build \
  --model_dir /models/llama-3.1-70b \
  --kv_cache_config kv_cache_config.json \
  --output_dir /trtllm_engines/llama-70b-nvme \
  --gemm_plugin fp8 \
  --max_batch_size 32 \
  --max_input_len 65536 \
  --max_seq_len 131072

NVMe optimization: io_parallelism, block_size alignment to SSD erase blocks

Set io_parallelism to match your NVMe queue depth, typically 4-8 for Gen4/Gen5 drives. Block size alignment is critical: 4KB alignment matches SSD erase block boundaries, reducing write amplification and improving throughput. Misaligned blocks can trigger read-modify-write cycles in the SSD controller, degrading performance by 30-50%.

Head-to-head comparison: 6 dimensions that decide your stack

Dimension	vLLM	TensorRT-LLM
NVMe offloading	Not native, CPU only	Native, optimized
Latency (long context)	Higher, decode stalls	Lower, prefetching
Throughput	Good for short context	Better at batch saturation
Multi-GPU setup	Simple tensor parallel	Requires explicit planning
Hardware support	CUDA + experimental AMD	CUDA only
Feature velocity	Rapid community updates	NVIDIA release cycle

Both engines support quantized models with offloading. Quantization reduces KV cache size, extending effective offloading capacity. TensorRT-LLM provides native quantized KV cache storage; vLLM requires compatible quantization config with attention backend support.

Latency under pressure: TTFT and TPOT degradation curves

Time to first token (TTFT) degrades linearly with context length for GPU-only inference. With NVMe offloading, TTFT shows step-function increases when cache misses trigger disk reads. TensorRT-LLM’s prefetching mitigates this by loading blocks asynchronously during token generation. vLLM’s CPU offloader shows latency spikes when the scheduler swaps blocks synchronously.

Throughput: requests per second at varying context lengths

vLLM’s continuous batching excels at high throughput for shorter contexts (under 16K tokens). TensorRT-LLM’s in-flight batching with NVMe offloading maintains throughput better at longer contexts. At 64K+ contexts, TensorRT-LLM typically shows 20-40% higher requests per second on equivalent hardware.

Memory efficiency: effective compression ratios

TensorRT-LLM’s enable_block_reuse achieves higher cache hit rates by reusing KV blocks across requests. vLLM’s enable_prefix_caching provides similar functionality but with less sophisticated eviction policies. Both reduce effective memory usage by 30-50% on workloads with repeated prompt prefixes.

Operational complexity: deployment and observability

vLLM exposes Prometheus metrics at the /metrics endpoint, covering cache hit rates, block usage, and request latencies. TensorRT-LLM integrates with Triton Inference Server, providing metrics through Triton’s Prometheus exporter. vLLM’s simpler deployment model reduces operational overhead; TensorRT-LLM requires engine compilation for each configuration change.

Ecosystem: quantization, speculative decoding, LoRA support

vLLM supports INT8/INT4 quantization via AWQ and GPTQ, speculative decoding, and LoRA adapters at runtime. TensorRT-LLM supports FP8/INT8 quantization with native kernels, speculative decoding through draft models, and LoRA via plugin architecture. Feature parity is close, but vLLM’s community-driven ecosystem often adds support for new techniques faster. For a deeper look at multi-step workflows with these models, see how to orchestrate multi-step llm workflows with node.js and crewai.

Hardware lock-in: CUDA-only vs experimental AMD support

TensorRT-LLM is CUDA-only, tied to NVIDIA hardware. vLLM has experimental ROCm support for AMD GPUs, though it is not production-ready as of late 2026 [VERIFY: Has AMD ROCm support for either engine matured to production viability?]. For on-premise deployments with mixed hardware, this is a critical consideration.

Decision matrix: choose your engine in 60 seconds

Use this decision tree based on your constraints:

1. Do you need NVMe offloading (context exceeds CPU RAM capacity)? Yes -> TensorRT-LLM
2. Is p99 TTFT under 100ms required at 64K+ context? Yes -> TensorRT-LLM
3. Are you deploying on non-NVIDIA hardware? Yes -> vLLM (with ROCm)
4. Do you need rapid iteration with LoRA adapters and speculative decoding? Yes -> vLLM
5. Default choice for balanced latency/throughput -> vLLM for simplicity, TensorRT-LLM for scale

Scenario A: 100K+ context, single A100, strict latency SLO

Choose TensorRT-LLM with NVMe offloading. Configure io_parallelism to 8, use Gen4/Gen5 NVMe with dedicated namespace, and set enable_block_reuse. Monitor NVMe queue depth to prevent saturation. Expect p99 TTFT of 150-300ms depending on SSD performance.

Scenario B: multi-tenant SaaS, variable lengths, cost optimization

Choose vLLM with CPU offloading and prefix caching. Set gpu_memory_utilization to 0.95 and swap_space to half of system RAM. Enable enable_prefix_caching for common system prompts. This maximizes throughput on shorter queries while handling occasional long requests with graceful degradation.

Scenario C: edge deployment, quantized models, CPU fallback

Choose vLLM for its broader hardware compatibility. Use INT4 AWQ quantization to fit models in smaller GPUs. Configure CPU offloading as fallback. This setup provides maximum flexibility for heterogeneous edge environments. Similar hybrid strategies appear when comparing a local llm (llama) vs. claude api for hybrid agents cost control.

Production configuration: working examples

vLLM: docker-compose with cpu_offload_gb and prometheus metrics

# docker-compose.yml
version: "3.8"
services:
  vllm-server:
    image: vllm/vllm-openai:v0.6.2
    container_name: vllm-server
    ports:
      - "8000:8000"
    environment:
      - MODEL_NAME=meta-llama/Llama-3.1-70B
      - TENSOR_PARALLEL_SIZE=2
      - GPU_MEMORY_UTILIZATION=0.90
      - SWAP_SPACE=64
      - ENABLE_PREFIX_CACHING=true
    volumes:
      - /models:/models
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: 2
              capabilities: [gpu]
    command: >
      --model /models/llama-3.1-70b
      --tensor-parallel-size 2
      --gpu-memory-utilization 0.90
      --swap-space 64
      --enable-prefix-caching
      --max-model-len 65536

  prometheus:
    image: prom/prometheus:v2.48.0
    ports:
      - "9090:9090"
    volumes:
      - ./prometheus.yml:/etc/prometheus/prometheus.yml

TensorRT-LLM: trtllm-build with kv_cache_config JSON

Build and deploy with explicit NVMe configuration:

#!/bin/bash
# build_trtllm_nvme.sh

MODEL_DIR="/models/llama-3.1-70b"
OUTPUT_DIR="/trtllm_engines/llama-70b-nvme"
KV_CONFIG="kv_cache_config.json"

trtllm-build \
  --model_dir $MODEL_DIR \
  --kv_cache_config $KV_CONFIG \
  --output_dir $OUTPUT_DIR \
  --gemm_plugin fp8 \
  --max_batch_size 32 \
  --max_input_len 65536 \
  --max_seq_len 131072 \
  --tokens_per_block 16

# Launch with Triton
tritonserver \
  --model-repository=/triton_model_repo \
  --grpc-port=8001 \
  --http-port=8000

Triton Inference Server integration for both

Both engines integrate with Triton Inference Server. vLLM uses the Python backend, while TensorRT-LLM uses the TensorRT backend. Configure Triton’s dynamic batching and rate limiting to prevent request bursts from overwhelming the offloading layer. For more on handling request spikes, see the ai gateway circuit breaker pattern for llm fallback routing.

Common errors and fixes

Error: ‘CUDA out of memory’ despite cpu_offload_gb set Cause: Attention metadata and activation tensors are not included in offloading accounting. The scheduler underestimates total memory requirements. Fix: Reduce gpu_memory_utilization to 0.80-0.85 to leave headroom for non-cache tensors. Monitor vllm:num_gpu_blocks metric.

Symptom: Periodic TTFT spikes >5s Cause: NVMe queue depth saturation. The SSD cannot process read requests fast enough during cache misses. Fix: Reduce batch size or increase io_parallelism. Monitor with nvme io-passthru /dev/nvme0n1 --opcode 0x0a or iostat -x 1. Lower queue depth thresholds.

Error: TensorRT-LLM ‘KV cache allocation failed’ Cause: Block size multiplied by max_blocks exceeds available NVMe space or filesystem limits. Fix: Verify NVMe capacity with df -h. Calculate required space: max_seqs * max_blocks * block_size * dtype_size * 2. Reduce max_batch_size or increase NVMe allocation.

Symptom: GPU utilization <50% with queued requests Cause: CPU-GPU synchronization bottleneck. The scheduler waits for block transfers, underutilizing GPU compute. Fix: Enable async prefetching in TensorRT-LLM. In vLLM, increase swap_space to reduce swap frequency. Check PCIe bandwidth with nvidia-smi topo -m.

Symptom: Gradual performance degradation over hours Cause: SSD thermal throttling or garbage collection pressure. Fix: Monitor nvme smart-log /dev/nvme0n1 for temperature and percentage used. Add NVMe cooling. Use enterprise-grade SSDs with sustained write ratings.

Failure modes and debugging checklist

Minimum hardware for KV cache offloading to NVMe includes PCIe Gen4 x4 NVMe SSD with sustained read >3 GB/s, 32+ GB system DRAM for metadata and buffering, and a GPU with sufficient compute to overlap I/O [VERIFY: Are there new PCIe Gen5 NVMe performance baselines that supersede cited Gen4 figures?]. Optimal configuration uses PCIe Gen5 x4, dedicated NVMe namespaces, and CPU with ample PCIe lanes.

Diagnostic commands

1. Check NVMe health and performance:

nvme smart-log /dev/nvme0n1
nvme get-log /dev/nvme0n1 --log-id=2 --log-len=512
iostat -xmt 1 /dev/nvme0n1

1. Monitor GPU memory and PCIe traffic:

nvidia-smi dmon -s puc -d 1
nvidia-smi topo -m
watch -n 1 nvidia-smi --query-gpu=memory.used,memory.total,utilization.gpu --format=csv

1. Check vLLM metrics:

curl http://localhost:8000/metrics | grep -E "vllm_(cache|requests|tokens)"

Monitoring checklist

Key metrics to monitor: GPU memory usage (nvidia-smi), NVMe read/write throughput and IOPS (nvme-cli smart-log or iostat), TTFT and TPOT percentiles (engine-specific metrics endpoints), and PCIe bandwidth utilization (nvidia-smi topo -m). Set alerts on queue depth saturation and GPU idle cycles during I/O.

Performance tuning: extracting maximum from NVMe

Optimal NVMe offloading requires 4KB-aligned block sizes, io_uring with sufficient queue depth, and kernel bypass for sub-millisecond access when available.

Block size selection: model hidden dim, head count, and SSD geometry

For Llama-3.1-70B with hidden_dim=8192 and 64 heads, each token requires 8192 * 64 * 2 * 2 = 2MB for FP16 KV cache. A block of 16 tokens