AI
Speculative decoding threshold tuning for code generation
Configure speculative decoding acceptance thresholds to optimize code generation throughput and prevent costly inference latency regressions.
// tag archive
llm-inference
4 articles
AI
RadixAttention memory layout strategies explained
Diagnose GPU OOM errors and low cache hit rates by comparing contiguous vs paged KV cache layouts for…
AI
Speculative Decoding LLaMA Speedup Measurement
Construct a reproducible benchmarking harness to measure speculative decoding speedup using wall-clock time and valid statistical sampling methods.
AI
RadixAttention prefix caching: vLLM latency reduction
Understand how RadixAttention prefix caching in vLLM eliminates redundant prefill computation to significantly reduce time-to-first-token latency.