TL;DR
NarrowSpec demonstrates that "feature-conditioned" draft models in speculative decoding—which typically inherit the target model's hidden width—can run at 1/2 or even 1/4 of that width with minimal impact on token acceptance. By shrinking the draft's residual stream and using a "folding" technique for output adapters, the authors achieve end-to-end decoding speedups of up to 1.44x on modern NVIDIA GPUs.
What Problem Does This Paper Address?
Speculative decoding (SD) is a standard technique to accelerate Large Language Model (LLM) inference. It uses a small, fast "draft" model to propose a sequence of tokens, which a large "target" model then verifies in a single parallel pass.
The state-of-the-art in drafting involves "feature-conditioned" models like DFlash and EAGLE-3. These models don't just look at previous tokens; they "tap" into the target model's internal hidden states to make better predictions. Because they read these $d_{target}$-wide vectors, researchers have traditionally designed these drafters to use the same hidden width as the target.
However, this makes the draft model computationally heavier than necessary. The authors of NarrowSpec ask: Does the draft's internal residual stream really need to be that wide? If we can narrow the draft without significantly hurting its ability to predict the target's output (the "acceptance length"), we can significantly reduce the per-token latency.
Methods
The researchers introduced NarrowSpec, a methodology for training and deploying narrow draft models:
- Narrowing the Residual Stream: They set the draft's hidden width ($d_d$) to $d_t/2$ or $d_t/4$. While the residual stream shrinks, they keep the attention head count and dimensions fixed to maintain the draft's expressive power in its attention mechanism.
- Width Adapters: To bridge the gap between the target's wide features and the draft's narrow stream, they use linear "input adapters" (projecting $d_t \to d_d$) and "output adapters" (projecting $d_d \to d_t$).
- LM-Head Folding: This is the "secret sauce" for efficiency. For drafters that borrow the target's vocabulary projection (like DFlash), they don't run the output adapter and the target's head sequentially. Instead, they mathematically "fold" the two linear operations into a single narrow projection matrix ($W_{fold} = W_{lm}W_{out}$) at load time. This ensures the logits remain virtually identical (up to floating-point rounding) while drastically reducing the number of parameters the draft needs to read and compute during inference.
The authors evaluated two distinct drafter families:
- DFlash: A block-parallel drafter that proposes multiple tokens in one forward pass.
- EAGLE-3: An autoregressive drafter that runs a single decoder layer multiple times.
Datasets
- Training: The drafts were trained on ~800K samples from Nemotron-Post-Training-v2 and CodeAlpaca, with responses regenerated by the target models at temperature 0.
- Evaluation:
- SPEED-Bench: An 880-prompt qualitative split spanning 11 categories (coding, math, RAG, etc.).
- Standard Benchmarks: GSM8K, MATH-500, AIME25, HumanEval, MBPP, LiveCodeBench, MT-Bench, and Alpaca.
Key Insights and Ideas
- Capacity Over-provisioning: Draft models are often over-provisioned. A residual stream at 1/4 width still captures enough "signal" from the target's hidden states to maintain high acceptance rates.
- The Bandwidth Bottleneck: The benefits of narrowing are highly dependent on the GPU's memory bandwidth. On the DGX Spark (GB10), which has lower bandwidth than the B200, narrowing provides a much larger relative speedup because the draft's smaller kernels stay closer to the hardware's peak bandwidth.
- Folding is Free Speed: Folding the output adapter into the LM head is a "no-brainer" optimization. It removes the latency of the adapter entirely and reduces the bytes read from the massive vocabulary projection matrix.
- Attention is the Floor: Since head geometry is kept constant, the cost of the attention mechanism and the KV cache does not shrink. This means that at extremely long context lengths, the relative gains from narrowing the residual stream will eventually diminish.
Key Results
The authors tested their approach on Qwen3-8B and GPT-OSS-20B (a 20B MoE model) across different hardware:
- End-to-End Speedups: On the B200, narrowing to 1/4 width provided 1.06–1.12x speedup. On the GB10, this jumped to 1.20–1.44x.
- Acceptance Trade-off: Halving the width resulted in a negligible drop in acceptance (often $<1\%$, max $3\%$). Quartering the width cost between 2.1% and 5.4% in acceptance length.
- Benchmark Stability: Across eight standard benchmarks, the narrow drafts maintained performance parity with full-width drafts in three out of four drafter-target pairs.
Limitations and Future Work
- Quantization Interaction: The authors noted that while narrowing and quantization (e.g., to NVFP4) are complementary, the marginal gains from quantization decrease as the model gets narrower.
- Context Length Limits: The study capped context at 32k tokens. Because attention costs don't scale down with width, the "narrowing dividend" may shrink at million-token scales.
- Training from Scratch: Currently, narrow drafts must be trained from scratch. Exploring whether a full-width draft can be "distilled" or "pruned" into a narrow one without full retraining remains an open question.
- Fixed Training Budget: The results are based on a fixed token budget rather than training to absolute convergence; different budgets might shift the optimal width-accuracy trade-off.
Summary updated