When Does Better Acceptance Pay? A Controlled Study of Speculative Decoding

Yongchao Long, Chunrui Zheng, Yuxuan Zhou, Guangkun Nie, Tian-Chi Ma, Chao Pang, Shuohuan Wang, Yuchen Ding, Xianbin Wen, Dianhai Yu, Junyuan Shang, Yuxi Zhou, Shenda Hong

Speculative decoding (SD) feels like a "free lunch" for LLM inference. By using a tiny "drafter" model to guess tokens and a giant "target" model to verify them in parallel, we can theoretically get the same output much faster. But as any engineer knows, there’s no such thing as a free lunch—the time spent drafting must be repaid by the tokens accepted.

In their recent paper, researchers from Baidu and several universities take a hard look at the current state of speculative decoding, asking a critical question: When is a higher acceptance rate actually worth the extra computational cost?

TL;DR

Most reported performance gaps between speculative decoding methods shrink significantly when training data and inference engines are aligned. The study finds that "nearly free" improvements to training objectives and context often yield better throughput than complex, high-latency drafter architectures, leading to a new strong baseline called DCore.

What Problem Does This Paper Address?

The field of speculative decoding is currently a bit of a "Wild West." New papers frequently claim state-of-the-art (SOTA) results, but they often use different training datasets, target models, and inference implementations. This makes it nearly impossible to tell if a 2x speedup is due to a brilliant new architecture or simply a better training corpus.

The authors address this lack of controlled comparison by establishing a rigorous protocol. They aim to disentangle the gains from proposal quality (how often the drafter is right) and execution cost (how long the drafter takes to run).

Methods

The researchers developed a Controlled Protocol consisting of two main stages:

  1. Re-measurement: Running released checkpoints from six popular drafter recipes in a common, optimized inference engine (SGLang).
  2. Re-training: Training all six recipes from scratch on the exact same target-generated data to ensure a level playing field.

The study covers three major "families" of drafters:

  • Autoregressive (AR): e.g., Eagle-3, which generates tokens one by one.
  • Block-parallel: e.g., DFlash, DFlare, D-PACE, which predict multiple future positions in a single pass.
  • Semi-autoregressive (semi-AR): e.g., Domino, DSpark, which add causal correction to parallel predictions.

They also introduced DCore, a "best-of-both-worlds" baseline that combines the low-latency backbone of DFlash with the superior context mixing of DFlare and the optimized training objective of D-PACE.

Datasets

  • Training: A shared corpus generated by Qwen3-4B and Qwen3-8B target models responding to the Open-PerfectBlend prompt set.
  • Evaluation: Ten diverse benchmarks covering math (GSM8K, MATH-500, AIME25), code (HumanEval, MBPP, LiveCodeBench), and chat/agentic tasks (MT-Bench, AlpacaEval, Arena-Hard, SWE-Bench).
  • SpecMix: A newly constructed mixture of 3,331 prompts from all ten benchmarks used to simulate realistic, multi-task serving traffic.

Key Insights and Ideas

  • Alignment Narrows the Gaps: This is perhaps the most striking finding. When you use the same training data and inference engine, the massive performance differences reported in original papers often evaporate. For example, on the GSM8K benchmark, the speedup gap between the fastest and slowest methods dropped from a reported 4.60x to just 1.53x after re-training.
Figure 1: Observed gaps narrow after inference and training alignment. Four drafters on GSM8K, repre- senting the low-entropy half of our benchmark suite, and MT-Bench, representing the high-entropy half, on Qwen3-8B. Pale bars: acceptance length τ (left axis); solid bars: speedup over the target-only baseline (right axis). As reported: each paper’s own published num- bers. Re-measured: its released checkpoint re-run in our engine. Re-trained: a drafter we train on the same training data.
  • The "Nearly Free" Route to Speed: You can improve acceptance rates in two ways: spending more compute (e.g., adding layers to the drafter) or improving the training objective and context. The authors found that "nearly free" changes—like the D-PACE objective—often provide better throughput gains because they don't increase drafting latency.
Figure 2: Design ablation roadmap showing the path to DCore.
  • Entropy is the Drafter's Kryptonite: Drafters struggle most when the target model is uncertain. The study shows a strong negative correlation (~ -0.85) between task entropy (uncertainty) and acceptance length. Interestingly, accepted token sequences tend to have much lower entropy than the "rejected" tokens that end a draft path.
Figure 3: Accepted paths tend to have lower target entropy than their rejected endpoints across three drafter families. Qwen3-8B at 1|0: Eagle-3 (chain), DFlash, and DSpark. Cells show mean raw-T=1 target entropy at draft position j; rows show committed tokens nc + 1, where nc is the number of accepted drafts. Black borders mark rejected positions; grey bars show the share of steps at each length (full strip 36%). B=16 is the main grid and B=8 the inset. Other targets and temperatures are in Fig. 11.
  • Serving Conditions Dictate the Winner: The "best" drafter depends on your load. At high concurrency (many requests at once), the system becomes compute-bound, and the overhead of a large verification budget (B=16) can actually decrease throughput compared to a smaller budget (B=8).

Key Results

  • DCore Dominance: The authors' DCore baseline achieved the highest mean throughput in 18 out of 24 single-request settings. It proved that a well-tuned, low-overhead drafter is extremely hard to beat.
  • Budget Elasticity: Increasing the verification budget from 8 to 16 tokens provides diminishing returns. While it can boost throughput by up to 20% on low-entropy tasks, it offers almost no gain (max 2.8%) on high-entropy tasks.
Figure 4: Budget elasticity: what a larger budget buys, and what it costs (Qwen3-8B, 1|0). We use log changes to express throughput gains as the sum of acceptance gains and efficiency changes, revealing whether additional acceptance repays its execution cost. Axes: log change in acceptance length τ and throughput from B=8 to 16. Eagle-3(tree) compares 7/4/8 → 7/10/16; Eagle-3(chain) compares 7/1/8 →15/1/16. Parallel guide labels give 100 ∆ln η via Eq. (1), not ordinary percentage changes. Negative TPS changes indicate throughput loss.
  • Efficiency Matters: For block-parallel drafters, "effective efficiency" (how well acceptance translates to speedup) stays high (~0.69), whereas more complex autoregressive drafters like Eagle-3 can see efficiency drop significantly due to their higher sequential drafting cost.

Limitations and Future Work

The authors acknowledge several limitations:

  • Model Scope: The study focused on the Qwen3 model family and two specific target sizes (4B and 8B). Results might differ for significantly larger models (e.g., 70B+) where drafting costs are even more negligible.
  • Adaptive Budgets: While they touched on task uncertainty, they did not deeply explore adaptive budget control policies that change the draft length on the fly based on real-time entropy.
  • Hardware Portability: Throughput is highly dependent on specific GPU kernels and hardware; while they used a common engine, results might shift on different hardware architectures.

In conclusion, this paper serves as a much-needed reality check for the speculative decoding community. It highlights that in the race for higher acceptance rates, we must never lose sight of the clock.

Summary updated

Abstract

The throughput gains of speculative decoding depend on the trade-off between acceptance length and execution latency. We ask when additional acceptance is worth the cost of obtaining it. We establish a comparison protocol spanning training, inference, and evaluation, remeasuring released checkpoints and retraining six drafter recipes on shared target-generated data across ten benchmarks and two target models. Alignment changes the picture: gaps that appear large in published comparisons shrink substantially under shared training and inference conditions. This narrowing motivates two deeper questions: how is acceptance improved at different costs, and when do these gains improve throughput? We find that acceptance can be improved through nearly free objective and context changes or more expensive token-to-token conditioning. DCore illustrates the former route as a strong baseline with little additional measured drafting overhead. The payoff is conditional: task uncertainty and serving conditions affect acceptance returns, execution costs, and the preferred drafter and budget. These findings inform future drafter design and deployment choices.
BibTeX
@misc{long2026when,
  author = {Yongchao Long and Chunrui Zheng and Yuxuan Zhou and Guangkun Nie and Tian-Chi Ma and Chao Pang and Shuohuan Wang and Yuchen Ding and Xianbin Wen and Dianhai Yu and Junyuan Shang and Yuxi Zhou and Shenda Hong},
  title = {{When Does Better Acceptance Pay? A Controlled Study of Speculative Decoding}},
  year = {2026},
  eprint = {2610.24993},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2610.24993},
}
Figure