Redis eviction policies for LLM semantic caches

You are burning money on LLM APIs because your semantic cache evicts the wrong embeddings. Frequently accessed prompts disappear from memory while one-shot queries consume valuable space, driving up token costs. Selecting the right Redis semantic cache eviction policies lets you align memory pressure with token consumption patterns, cutting API spend without cache thrashing. By the end of this guide, you will be able to configure Redis to preserve high-value embeddings and size your memory to match your LLM token budget.

What is a semantic cache and why do LLM applications need eviction policies?

A semantic cache stores vector embeddings and their corresponding LLM responses, enabling retrieval of similar past queries without API calls. Unlike exact-match caches, it uses similarity thresholds, making eviction policy choice critical because approximate matches decay in value faster than exact hits.

How embedding storage differs from key-value caching

Traditional caching relies on exact key matches. You query user:123:profile, you get the exact bytes stored. Semantic caching changes this model. Instead of matching strings, you match mathematical representations. A user query about “database performance tuning” generates a 1536-dimension vector. The cache must find stored vectors within a cosine similarity threshold, even if the original prompt differed.

This approximate matching creates unique memory constraints. An exact key either exists or it does not. A semantic match exists on a spectrum. You might have ten similar queries cached, but only one is close enough to return. The other nine occupy memory that could store genuinely useful embeddings.

Why token budgets create unique memory pressure patterns

LLM API costs scale with token volume. Every cache miss triggers a full API round-trip, consuming input and output tokens. A single GPT-4 response can cost $0.03 to $0.60 depending on length [openai.com]. Your cache must maximize hit rate on expensive queries while aggressively evicting cheap duplicates.

Redis scales vertically to approximately 500 million 768-dimension embeddings per node with HNSW. Sharding via Redis Cluster enables billion-scale with cross-node similarity search tradeoffs. Memory, not CPU, is the binding constraint. Every gigabyte of Redis memory you allocate either serves high-value cache hits or wastes space on stale embeddings.

How do Redis eviction policies work with vector data in RedisVL?

Redis eviction policies apply to the underlying keyspace, but RedisVL adds index-level semantics. The maxmemory-policy directive determines which entries survive when RAM hits limits, while vector indexes using HNSW or FLAT require contiguous memory that eviction can fragment.

RedisVL index memory vs. raw key memory accounting

When you store an embedding in RedisVL, two things occupy memory. First, the raw vector bytes plus metadata in a Redis hash. Second, the HNSW index structure that enables fast approximate search.

HNSW index memory is significant. For configuration with M=16 and EF=200, the index structure consumes approximately 2x the raw vector size [arxiv.org]. A repository storing 1 million 1536-dimension embeddings allocated 6GB for raw vectors might need 12GB for the HNSW index alone.

This overhead changes your eviction math. Evicting a key deletes the hash. The HNSW index node persists until compaction or explicit index rebuild. Aggressive eviction can actually increase memory fragmentation as index structures orphan.

RedisJSON stores structured response data. RedisVL provides vector search capabilities. Production caches use both: RedisVL for similarity lookup, RedisJSON for response payload with joint eviction via key prefixes.

maxmemory-policy interaction with search indices

The standard Redis maxmemory-policy setting controls behavior when memory fills. For LLM semantic caches, three policies matter most.

allkeys-lru evicts the least recently accessed key regardless of TTL. This works well for embedding caches where access frequency predicts future hits. Recent queries often repeat.

volatile-ttl evicts keys with the shortest remaining TTL first. This requires you to set expiration times. It protects non-expiring keys but risks evicting high-value entries with short TTLs.

noeviction rejects writes when memory fills. LLM APIs fail open to direct calls, losing the cache benefit entirely. This policy suits only workloads with perfectly predictable working sets.

volatile-ttl vs. allkeys-lRU for embedding entries

Use LRU for user-facing queries with unpredictable patterns. Use TTL for session-bound contexts. The best production approach combines both: LRU base with TTL ceiling prevents stale semantic matches from accumulating.

HNSW index size directly affects eviction behavior. HNSW index memory is non-trivial for typical configurations. Eviction of underlying hashes does not reclaim HNSW nodes immediately. Periodic index compaction or explicit FT.DROPINDEX recreation is needed to reclaim wasted memory.

Which eviction policy maximizes cache hits within a fixed token budget?

Allkeys-lru with per-entry TTL metadata wins for LLM workloads. It preserves recently accessed embeddings (likely to re-query) while TTL boundaries prevent stale semantic matches from consuming budget. Volatile-ttl alone risks evicting high-value recent entries for arbitrary expirations.

Token-cost-weighted LRU

Standard LRU tracks only recency. It treats a $0.001 embedding lookup identically to a $0.50 GPT-4 response. For LLM caches, you want to weight eviction decisions by token cost saved.

You can implement custom scoring by storing cost metadata with each entry. Use RedisJSON to persist the original token count, model used, and timestamp. A Lua script reads this metadata during eviction pressure and makes intelligent decisions.

For example, evict the oldest entry with token cost under a threshold. Preserve expensive responses even if they were accessed less recently. This requires noeviction policy with background pruning rather than automatic eviction.

Hybrid noeviction with background TTL pruning

The hybrid approach uses noeviction in Redis configuration. A background process runs every N seconds. It queries keys with TTL thresholds and manually deletes candidates.

This gives you complete control over eviction logic. You can implement rules like: evict entries older than 24 hours with zero access count, or evict low-confidence semantic matches first.

Implement similarity threshold floors (cosine greater than 0.85) in RedisVL queries and store confidence scores. Low-confidence entries get volatile-ttl with short expiration, isolating noise from valuable cache real estate.

The tradeoff is complexity. You must monitor memory manually and handle OOM gracefully. A simple implementation using RedisGears 2.0 triggers on memory thresholds and executes custom logic.

When to accept OOM vs. eviction tradeoffs

Accept OOM risk only when your working set is predictable and bounded. If you know your daily unique query count and embedding model, you can size Redis to hold 100 percent plus buffer.

Never use noeviction with unbounded growth patterns. User-generated prompts, exploratory API usage, or multi-tenant workloads require automatic eviction to prevent service disruption.

The table below summarizes the tradeoffs.

Policy	Best For	Risk	Token Budget Impact
allkeys-lru	Unpredictable query patterns	Evicts expensive rare entries	Maximizes hit rate for repeating queries
volatile-ttl	Session-based contexts	Misclassified TTL causes loss	Controls semantic drift via expiration
noeviction	Bounded, predictable sets	OOM crashes entire cache	Guarantees consistency for fixed working sets
allkeys-lru + TTL hybrid	Production LLM workloads	Complex implementation	Optimal: recent + expensive preserved

How do you size Redis memory for token-budget constraints?

Size Redis at 1.5x your working set of unique embeddings plus response buffer. Calculate working set as (daily unique queries multiplied by average embedding size multiplied by 2 for HNSW overhead) divided by desired cache coverage ratio. Token savings equal cache hit rate multiplied by average tokens per API call multiplied by cost per 1K tokens.

Calculating embedding memory footprint per model

Different embedding models produce different vector dimensions. Larger dimensions improve similarity accuracy but consume more memory.

Model	Dimensions	Bytes per Vector	1M Vectors Memory
OpenAI text-embedding-3-small	1536	6,144	6 GB
OpenAI text-embedding-3-large	3072	12,288	12 GB
Cohere embed-english-v3.0	1024	4,096	4 GB
sentence-transformers/all-MiniLM-L6-v2	384	1,536	1.5 GB

[VERIFY: Current OpenAI embedding-3 pricing and dimension options]

OpenAI embeddings use float32 storage (4 bytes per dimension). The raw vector memory is dimensions multiplied by 4. Redis hash overhead adds approximately 100 bytes per key for metadata.

HNSW index overhead multipliers by M and EF parameters

HNSW configuration directly impacts memory. The M parameter controls the number of connections per node. Higher M improves recall but increases memory linearly.

For M=16, plan for 2x raw vector memory. For M=32, plan for 3-4x. The EF_CONSTRUCTION parameter affects index build time and quality but not persistent size. EF_RUNTIME affects query performance, not memory.

A production rule: set M to the square root of your expected nearest neighbors. For semantic search returning the top 10 matches, M=16 is sufficient. For top 100 matches, use M=32.

Token-price to Redis-GB cost equivalence

Cloud Redis (AWS ElastiCache, Redis Cloud) costs approximately $0.50 to $2.00 per GB per month depending on tier. OpenAI GPT-4 Turbo costs $0.01 per 1K input tokens and $0.03 per 1K output tokens.

A single cache hit saves the full API call cost. If your average response is 500 output tokens, each hit saves $0.015. To break even on 1 GB of Redis memory:

Redis cost: $1.00/GB/month
Required hits: $1.00 / $0.015 = 67 cache hits per GB per month

If 1 GB stores 100,000 embeddings and you achieve 5 percent hit rate on those embeddings, you get 5,000 hits per month per GB. That is 75x the break-even threshold. Semantic caching is almost always cost-effective for LLM workloads.

How do you implement token-aware cache warming and preemption?

Warm caches with high-probability prompts from analytics logs using RedisGears or server-side functions. Preempt low-value entries by storing token-cost metadata in RedisJSON and querying before eviction via custom module or Lua scripts with redis-server 7.2+.

[VERIFY: RedisGears 2.0 vs. new triggers and functions architecture in Redis 7.4]

RedisGears 2.0 functions for predictive warming

RedisGears 2.0 enables server-side JavaScript and Python functions that execute on triggers. You can register a function that runs daily, reads access logs from an external source, and populates the cache with predicted hot queries.

For example, analyze yesterday’s top 1000 queries. Generate embeddings for each. Store in RedisVL with allkeys-lru policy. Set TTL based on query volatility. Frequently identical prompts get long TTLs. Highly variable prompts get short TTLs.

This proactive warming ensures your cache starts each day with relevant data. The alternative, lazy population, suffers cold-start penalties during peak hours.

Lua-based custom eviction with cost metadata

Deploy a Lua script that evaluates eviction candidates by token cost. The script runs when memory usage exceeds a threshold.

-- File: evict_by_cost.lua
-- Evicts low-value entries when memory pressure exceeds threshold

local memory_usage = tonumber(redis.call('INFO', 'memory')['used_memory'])
local max_memory = tonumber(ARGV[1])  -- Pass max memory in bytes
local cost_threshold = tonumber(ARGV[2])  -- Minimum token cost to preserve

if memory_usage < max_memory * 0.9 then
    return {ok = "memory_ok", evicted = 0}
end

-- Find candidate keys for eviction
local candidates = redis.call('KEYS', 'llm_cache:*')
local evicted = 0

for _, key in ipairs(candidates) do
    local cost = tonumber(redis.call('HGET', key, 'token_cost') or 0)
    local last_access = tonumber(redis.call('HGET', key, 'last_access') or 0)
    local now = tonumber(ARGV[3])
    
    -- Evict if low cost AND older than 1 hour
    if cost < cost_threshold and (now - last_access) > 3600 then
        redis.call('DEL', key)
        evicted = evicted + 1
    end
    
    -- Stop if memory pressure relieved
    if redis.call('INFO', 'memory')['used_memory'] < max_memory * 0.8 then
        break
    end
end

return {ok = "evicted", count = evicted}

Call this script from your application when INFO memory shows usage above your target threshold.

Integration with OpenRouter or LiteLLM usage tracking

LiteLLM proxy configuration for multi-provider caching logs token usage across multiple providers. You can pipe these logs to Redis for cache decision feedback.

When LiteLLM logs a completion, extract the prompt hash, token count, and model. Write to a Redis stream. A background consumer reads the stream and updates cache metadata (hit rate, cost saved, last access time).

This telemetry closes the loop between cache behavior and actual cost savings. You can visualize ROI by observing the delta between cached responses and API calls.

What monitoring signals indicate eviction policy misconfiguration?

Watch for eviction rate greater than 5 percent of operations per second alongside cache hit rate below 60 percent for similar queries. This signals LRU decay faster than query re-access patterns. Track redis_keyspace_misses spikes with flat LLM API costs, which indicates semantic drift or threshold too strict.

Redis INFO metrics that correlate with token waste

The INFO memory section reveals immediate pressure.

• used_memory: Current total usage
• used_memory_peak: Highest usage since startup
• mem_fragmentation_ratio: Ratio of allocated to used memory. Above 1.5 indicates fragmentation.

The INFO stats section shows cache effectiveness.

• keyspace_hits: Number of successful lookups
• keyspace_misses: Number of failed lookups
• evicted_keys: Number of keys removed by eviction policy

Calculate cache hit ratio as keyspace_hits / (keyspace_hits + keyspace_misses). For LLM semantic caches, 60 to 80 percent is typical. Below 50 percent suggests your similarity threshold is too strict or your eviction policy is too aggressive.

RedisVL-specific index memory fragmentation alerts

RedisVL creates internal index keys with prefix __redisvl__. Monitor memory usage of these keys separately.

from redis import Redis
from redisvl.index import SearchIndex

# Connect to Redis
client = Redis.from_url("redis://localhost:6379")

# Get index memory usage
index_name = "llm_cache_index"
index_info = client.execute_command("FT.INFO", index_name)

# Parse memory statistics
for entry in index_info:
    if isinstance(entry, list) and len(entry) >= 2:
        key, value = entry[0], entry[1]
        if key == "num_records":
            print(f"Index contains {value} vectors")
        elif key == "bytes_per_record":
            print(f"Average {value} bytes per record")

[VERIFY: RedisVL 0.3.0 still current or has 0.4.0 shipped as of 2026-10-13?]

Unexpected growth in bytes_per_record indicates index bloat from incomplete eviction.

LLM provider cost variance as cache effectiveness signal

Your LLM API bill is the ultimate cache metric. Track daily cost variance. If costs spike without corresponding traffic increase, your cache is failing.

Integrate fine-grained AI cost attribution to correlate specific query patterns with cache misses. High-cost queries that miss repeatedly indicate warming opportunities. Low-cost queries that consume cache space indicate eviction candidates.

Prerequisites

To follow along with the implementation, you need:

• Redis Stack 7.4 or later with RedisVL module enabled (download from redis.io)
• Python 3.10 or later
• redis-py version 5.0.0 or later: pip install redis==5.0.0
• redisvl version 0.3.0 or later: pip install redisvl==0.3.0
• OpenAI API key for embedding generation (optional, for full implementation)
• Basic familiarity with vector similarity concepts

Last verified: 2026-10-13

Implementation

1. Setting up the RedisVL semantic cache

The first step is configuring RedisVL with a schema that supports token-cost metadata. This enables both similarity search and cost-aware eviction logic.

# File: semantic_cache_setup.py
import os
from redisvl.schema import IndexSchema
from redisvl.index import SearchIndex
from redis import Redis

# Define schema with token cost metadata
schema = IndexSchema.from_dict({
    "index": {
        "name": "llm_cache",
        "prefix": "cache:",
        "key_separator": ":"
    },
    "fields": [
        {"name": "prompt", "type": "text"},
        {"name": "response", "type": "text"},
        {
            "name": "embedding",
            "type": "vector",
            "attrs": {
                "algorithm": "hnsw",
                "datatype": "float32",
                "dims": 1536,
                "distance_metric": "cosine"
            }
        },
        {"name": "token_cost", "type": "numeric"},
        {"name": "model_name", "type": "tag"},
        {"name": "created_at", "type": "numeric"},
        {"name": "access_count", "type": "numeric"}
    ]
})

# Connect to Redis
client = Redis.from_url(os.environ.get("REDIS_URL", "redis://localhost:6379"))

# Create the index
index = SearchIndex(schema, client)
index.create(overwrite=True)

print(f"Index created: {index.name}")

Expected output:

Index created: llm_cache

2. Implementing cost-aware cache writes

Store embeddings with metadata that tracks token cost and access frequency. This data feeds the eviction decision logic.

# File: cache_write.py
import time
import hashlib
from redis import Redis
from redisvl.index import SearchIndex
from openai import OpenAI

client = Redis.from_url("redis://localhost:6379")
index = SearchIndex.from_name("llm_cache", client)
openai_client = OpenAI(api_key=os.environ.get("OPENAI_API_KEY"))

def get