Speculative Decoding LLaMA Speedup Measurement

Speculative decoding speedup claims of 2-3x are common, but you have likely seen inconsistent gains, negative speedups, or results that fail to replicate across hardware. You cannot tell if the culprit is your draft model, your measurement method, or a subtle bug in your implementation. A valid speculative decoding LLaMA speedup measurement requires wall-clock time-per-request across statistically significant samples, not theoretical token counts. This guide builds a reproducible benchmarking harness using vLLM 0.5.4+ and PyTorch 2.4+ so you can defend your speedup numbers in production.

What is speculative decoding and why does measurement methodology matter?

Speculative decoding accelerates LLM inference by having a small draft model generate candidate tokens that a larger target model verifies in parallel. Correct speedup measurement requires wall-clock time comparison, not theoretical token counts, because verification overhead, memory bandwidth, and draft model latency dominate real performance. If you measure throughput in tokens per second without controlling for these factors, your results will mislead.

The draft-then-verify mechanism: token generation vs. acceptance

In standard autoregressive decoding, each token requires a full forward pass through the model, creating a memory-bandwidth bound latency bottleneck. Speculative decoding breaks this by using a smaller “draft” model to propose multiple tokens, then verifying them in a single target model forward pass arxiv.org (Leviathan et al., Google Research). If the target model accepts a token, you get it “for free” without an additional target pass. If rejection occurs, the draft model’s work was wasted, but the output remains mathematically identical to standard decoding.

Published results show 1.5-2.5x speedup on LLaMA models when draft models are 7-10x smaller than targets and achieve 60-80% token acceptance rates. Actual speedup depends heavily on hardware (memory bandwidth vs. compute bound), batch size, and prompt distribution docs.vllm.ai.

Why throughput (tokens/sec) misleads: latency-sensitive vs. batch throughput

Tokens-per-second throughput metrics assume a fully saturated GPU, which rarely happens in latency-sensitive serving scenarios. Single-request latency is dominated by memory bandwidth for loading weights, not compute. A draft model reduces this bandwidth bottleneck by accepting tokens without target model weight reloads.

Publishing a single “speedup factor” without context hides critical variance. A 2x median speedup with a P99 slowdown is a production incident waiting to happen. You need to measure wall-clock time per request, not aggregate throughput, to understand real user impact.

Common measurement anti-patterns: ignoring warmup, single-sample benchmarks, CPU-GPU sync points

Bad measurements share three patterns. First, ignoring CUDA graph warmup: the first few inferences compile kernels and build CUDA graphs, causing 10-100x higher latency. Second, single-sample benchmarks: one request tells you nothing about variance from thermal throttling or OS scheduling. Third, missing CPU-GPU synchronization: timing CPU launch calls instead of GPU completion gives artificially low numbers.

Common causes of speculative decoding being slower than no speculation include low acceptance rate (<50%) from distribution mismatch or excessive temperature, memory bandwidth saturation from loading two models, small batch sizes where verification overhead dominates, or failure to use CUDA graphs causing CPU launch overhead for dual-model execution.

How do you set up a draft LLaMA model for speculative decoding?

Effective draft models are typically 7-10x smaller than target models, share tokenizer vocabulary, and run at higher temperature to increase token diversity and acceptance probability. The draft model size directly impacts the acceptance rate versus inference overhead trade-off.

Draft model selection criteria: size, architectural compatibility, domain alignment

Empirically, draft models 1/7th to 1/10th the size of the target model balance acceptance rate against inference overhead. For LLaMA-3.1 70B, LLaMA-68M or 160M are common choices. For LLaMA-3.1 8B, a 1B draft or n-gram lookahead may be preferable.

The draft model must share the exact tokenizer vocabulary with the target. Tokenizer mismatches cause silent acceptance failures because draft tokens map to different embeddings. Validate this with an assertion:

assert draft_tokenizer.vocab == target_tokenizer.vocab, \
    "Draft and target tokenizers must have identical vocabularies"

github.com/meta-llama provides official LLaMA model architectures and tokenizer specifications.

Domain alignment matters. A code-trained draft model will have poor acceptance rates when drafting for a general-purpose target model. Use a draft model trained on similar data distributions, or accept lower speedups.

Temperature tuning for draft models: exploration vs. acceptance rate trade-off

Draft model temperature controls the exploration-exploitation balance. Lower temperature (0.0-0.3) makes the draft model more confident but conservative, which matches greedy target decoding well but may miss creative completions. Higher temperature (0.7-1.0) increases token diversity, potentially improving acceptance for creative domains but reducing acceptance for factual or code prompts.

A practical starting point is draft temperature = target temperature + 0.2. Sweep temperatures from 0.0 to 1.0 in 0.1 increments while measuring acceptance rate to find the optimal point for your workload.

N-gram speculative decoding as a draft-free alternative: when to use it

N-gram lookahead decoding [VERIFY: vLLM 0.5.4 is current as of 2026-10-13; check for 0.6.0+ with specdecode API changes] uses cached n-grams from context instead of a draft model. This requires zero additional memory or model loading, but caps speedup at approximately 1.3x. Use n-gram lookahead when:

1. GPU memory is too tight for a second model.
2. Prompt distributions are highly repetitive (code completion, template filling).
3. You need deterministic latency without draft model variance.

Learned draft models enable 2x+ speedup but double memory footprint. N-gram requires no model at all.

What hardware and software stack do you need for reproducible benchmarks?

Reproducible measurement requires CUDA 12.1+, PyTorch 2.3+, and vLLM 0.5.0+ or Transformers 4.40+ with use_cache=True, CUDA graph capture enabled, and isolated GPU clocks to eliminate thermal throttling variance.

GPU isolation and clock locking: nvidia-smi -lgc for stable clocks

GPU clock frequencies fluctuate based on temperature and power limits, introducing variance into timing measurements. Lock clocks to a fixed frequency:

# Lock GPU clocks to maximum stable frequency
sudo nvidia-smi -i 0 -lgc 1410

Replace 1410 with your GPU’s maximum supported clock from nvidia-smi -q -d CLOCK. Run benchmarks with locked clocks, then release them:

sudo nvidia-smi -i 0 -rgc

This prevents thermal throttling during long benchmarks from contaminating measurements. Monitor temperature with nvidia-smi dmon to ensure you stay below thermal limits.

Memory bandwidth saturation detection: when PCIe or HBM becomes the bottleneck

Speculative decoding loads two models into GPU memory. If the combined model size exceeds HBM capacity, you get CUDA out of memory errors. If it approaches HBM limits, memory bandwidth becomes the bottleneck.

Detect bandwidth saturation by monitoring memory utilization:

nvidia-smi dmon -s mu -d 1

If memory utilization stays above 95% during inference, you are bandwidth-bound. Consider pipeline parallelism with draft and target on separate GPUs, or quantization to reduce memory pressure.

vLLM vs. Hugging Face Transformers: feature parity and measurement differences

vLLM provides optimized speculative decoding with built-in acceptance rate logging. Transformers requires manual implementation or patching of the generate loop for detailed metrics.

vLLM example with speculative decoding:

from vllm import LLM, SamplingParams

llm = LLM(
    model="meta-llama/Llama-3.1-70B",
    speculative_model="meta-llama/Llama-68M",
    num_speculative_tokens=5,
    tensor_parallel_size=4
)
sampling_params = SamplingParams(temperature=0.0, max_tokens=100)
outputs = llm.generate(["Hello, world!"], sampling_params)

Transformers requires manual instrumentation for acceptance rate tracking. vLLM exposes this via metrics logging when speculative decoding is enabled.

Prerequisites

• Hardware: NVIDIA GPU with CUDA 12.1+ support and 40GB+ VRAM (for 70B target + draft)
• Software: Ubuntu 22.04+ or containerized environment with Docker
• Python packages (pinned versions):

torch==2.4.1
vllm==0.5.4
transformers==4.44.2
numpy==1.26.4
scipy==1.13.1

Install requirements:

pip install torch==2.4.1 vllm==0.5.4 transformers==4.44.2 numpy==1.26.4 scipy==1.13.1

Last verified: 2026-10-13

How do you build a statistically valid speedup measurement harness?

A valid harness runs 1000+ requests with representative prompt distributions, measures end-to-end wall-clock time with torch.cuda.synchronize(), reports median and P99 latency with confidence intervals, and correlates speedup with per-request acceptance rates.

Warmup protocol: CUDA graph building, cache priming, thermal stabilization

The warmup phase eliminates one-time costs from measurements. vLLM builds CUDA graphs on first inference. Run warmup requests until latency stabilizes:

import torch
import time

def warmup_model(llm, sampling_params, num_warmup=10):
    """Run warmup inferences to stabilize CUDA graphs and cache."""
    warmup_prompts = ["Warmup prompt"] * num_warmup
    for _ in range(3):
        llm.generate(warmup_prompts, sampling_params)
    torch.cuda.synchronize()
    print("Warmup complete")

Measure temperature before and after warmup. If temperature continues rising during benchmarks, thermal throttling is contaminating results.

Measurement loop: time.perf_counter() with proper GPU synchronization

Use time.perf_counter() for wall-clock timing. Call torch.cuda.synchronize() before stopping the timer to ensure GPU work is complete:

def measure_inference(llm, prompts, sampling_params, num_runs=1000):
    """Measure inference latency with proper GPU synchronization."""
    import numpy as np
    
    latencies = []
    acceptance_rates = []
    
    for _ in range(num_runs):
        torch.cuda.synchronize()
        start = time.perf_counter()
        
        outputs = llm.generate(prompts, sampling_params)
        
        torch.cuda.synchronize()
        end = time.perf_counter()
        
        latencies.append(end - start)
        # Extract acceptance rate from vLLM metrics if available
        if hasattr(outputs[0], 'metrics'):
            acceptance_rates.append(outputs[0].metrics.get('acceptance_rate', 0))
    
    return {
        'median_latency': np.median(latencies),
        'p99_latency': np.percentile(latencies, 99),
        'mean_latency': np.mean(latencies),
        'std_latency': np.std(latencies),
        'acceptance_rates': acceptance_rates
    }

Running this for both speculative and baseline configurations gives comparable measurements.

Statistical rigor: bootstrapped confidence intervals, outlier detection, sample size power analysis

Report confidence intervals, not just point estimates. Bootstrap resampling provides robust intervals without distributional assumptions:

from scipy import stats

def bootstrap_ci(data, confidence=0.95, n_bootstrap=10000):
    """Calculate bootstrapped confidence interval for the mean."""
    bootstrap_means = np.random.choice(data, (n_bootstrap, len(data)), replace=True).mean(axis=1)
    lower = np.percentile(bootstrap_means, (1 - confidence) / 2 * 100)
    upper = np.percentile(bootstrap_means, (1 + confidence) / 2 * 100)
    return (lower, upper)

Detect outliers using the interquartile range method. Values beyond 1.5 * IQR from Q1 or Q3 warrant investigation. Exclude them only if you identify a specific cause (e.g., cold start after pause).

Sample size of 1000+ provides statistical power to detect 5% speedup differences with 80% power for typical latency variance.

What metrics should you track beyond wall-clock speedup?

Complete analysis requires token acceptance rate distribution, draft model tokens generated versus accepted ratio, verification batch efficiency, and memory overhead to enable root-cause analysis when speedups underperform.

Acceptance rate heatmaps: position-dependent degradation

Acceptance rate varies by token position. Early tokens often have higher acceptance because the draft model has more context. Later tokens show degradation as sequences diverge.

Instrument the verification loop to count accepted tokens versus generated draft tokens per request. Log per-position acceptance. vLLM exposes this via speculative metrics when enabled. For Transformers, patch the generate loop:

# Track acceptance by position
acceptance_by_position = defaultdict(list)

def log_acceptance(draft_tokens, accepted_mask):
    """Log acceptance rate by token position."""
    for pos, accepted in enumerate(accepted_mask):
        acceptance_by_position[pos].append(float(accepted))

Plot acceptance rate versus position to identify where draft quality deteriorates. This guides draft model temperature tuning or suggests when to adjust num_speculative_tokens.

Draft model inference time vs. target model verification time decomposition

Speedup breaks down into draft inference time, target verification time, and accepted token savings. Profile each component:

class TimingContext:
    def __init__(self):
        self.timings = {}
    
    def measure(self, name):
        """Context manager for timing code blocks."""
        class Timer:
            def __init__(ctx, name):
                ctx.name = name
            def __enter__(ctx):
                torch.cuda.synchronize()
                ctx.start = time.perf_counter()
                return ctx
            def __exit__(ctx, *args):
                torch.cuda.synchronize()
                ctx.end = time.perf_counter()
                self.timings[ctx.name] = ctx.end - ctx.start
        return Timer(name)

If draft inference time exceeds the time saved by accepted tokens, you have negative speedup. Either reduce draft model size or improve acceptance rate.

Memory bandwidth overhead: KV cache duplication costs

Two models mean two KV caches. Monitor memory allocation:

watch -n 0.5 nvidia-smi --query-gpu=memory.used,memory.total --format=csv

KV cache for the draft model adds memory overhead that can push you into bandwidth saturation. If memory overlaps with compute from other processes, latency spikes.

How do you interpret and debug suboptimal speedup results?

Negative or minimal speedups typically stem from low acceptance rates (<60%), draft model latency exceeding gains, memory bandwidth saturation from dual-model loading, or verification overhead from small batch sizes.

Diagnostic checklist: acceptance rate thresholds by draft-target size ratio

Use this decision tree for diagnosis:

flowchart TD
    A[Speedup < 1.2x?] --> B{Acceptance rate?}
    B -->|< 50%| C[Check domain alignment and temperature]
    B -->|50-60%| D[Tune draft temperature or adjust num_speculative_tokens]
    B -->|> 60%| E{Draft latency?}
    E -->|> 30% target latency| F[Reduce draft model size or use n-gram]
    E -->|< 30% target latency| G{Memory bandwidth?}
    G -->|> 90% utilized| H[Quantize models or use pipeline parallelism]
    G -->|< 90% utilized| I[Check CUDA graph capture and batch size]

Acceptance rate thresholds vary by draft-target size ratio. A 1/10 size draft needs 65%+ acceptance to break even. A 1/7 size draft needs 55%+.

When speculative decoding hurts: memory-bound regimes and small batch sizes

Speedup degrades when batch sizes are small because verification overhead dominates. Each verification pass has fixed cost regardless of batch size. For batch sizes under 4, speculative decoding often hurts.

Memory-bound regimes occur when both models fit in memory but approach capacity. HBM bandwidth becomes the limiting factor, and adding draft model memory traffic hurts more than parallel verification helps.

Can speculative decoding work with quantized models? Yes, but draft and target must use compatible quantization schemes. GPTQ/AWQ draft with FP16/FP8 target often hurts acceptance due to distribution shift. Best practice: quantize both identically or use draft FP16 with target INT8/FP8, with acceptance rate monitoring to detect degradation.

Calibration workflow: temperature sweep to maximize acceptance-efficiency product

Optimize the product of acceptance rate and inference efficiency. Run a temperature sweep on your prompt distribution:

def temperature_sweep(llm, prompts, temperatures, num_samples=100):
    """Sweep draft temperatures to find optimal acceptance-efficiency point."""
    results = {}
    for temp in temperatures:
        sampling_params = SamplingParams(temperature=temp, max_tokens=100)
        metrics = measure_inference(llm, prompts, sampling_params, num_samples)
        acceptance = np.mean(metrics['acceptance_rates'])
        # Efficiency = acceptance / (draft overhead factor)
        efficiency = acceptance / (1 + 0.1)  # Approximate overhead
        results[temp] = {
            'acceptance': acceptance,
            'efficiency': efficiency,
            'latency': metrics['median_latency']
        }
    return results

Select temperature that maximizes efficiency, not just acceptance. High acceptance with slow draft inference loses to moderate acceptance with fast draft.

Advanced: Multi-draft speculative decoding and tree attention

Multi-draft speculation with tree-based verification (e.g., Medusa, EAGLE) increases acceptance rates to 70-85% but requires careful measurement of tree construction overhead and memory amplification.

Medusa/EAGLE integration: training-free vs. learned draft heads

Medusa github.com/FasterDecoding adds multiple prediction heads to the target model, eliminating separate draft model loading. Each head predicts tokens at different offsets. This is training-free for inference but requires fine-tuning the heads.

EAGLE github.com/SafeAILab uses a learned draft model with tree-structured verification. [VERIFY: EAGLE-3 claims and whether training artifacts are available]. EAGLE-2 demonstrates 10-15% acceptance improvement over baseline speculative decoding.

Both approaches increase memory overhead but eliminate draft model loading latency. Measure total end-to-end latency, not just acceptance rate.

Tree attention verification: