TL;DR
TVP (Total Variation Power) is a new training objective for speculative decoding that solves the "vanishing gradient" problem inherent in standard Total Variation (TV) distance. By annealing from a coverage-focused loss to an acceptance-focused one, TVP consistently outperforms cross-entropy and complex specialized objectives, delivering up to a 12.9% improvement in acceptance length across various LLM architectures.
What Problem Does This Paper Address?
Speculative decoding is a popular technique to speed up Large Language Model (LLM) inference. It uses a small, fast "drafter" model to propose a sequence of tokens, which a large "target" model then verifies in parallel. The efficiency of this process hinges on the Expected Acceptance Length (EAL)—the average number of tokens the target model accepts from the drafter's proposal.
Theoretically, to maximize EAL, the drafter should minimize the Total Variation (TV) distance between its distribution and the target's. However, TV has a major flaw: its gradient vanishes when the drafter puts very little probability on a token the target model likes (an "undercovered" token). This makes it nearly impossible to train a drafter from scratch using pure TV.
On the other hand, directly optimizing for EAL leads to "prefix attenuation." Because a token can only be accepted if all preceding tokens were also accepted, the training signal for later positions in a draft becomes incredibly weak. Most researchers fall back on Cross-Entropy (CE) or KL divergence, which provide better gradients but don't directly optimize for the acceptance mechanism used during inference.
Methods
The authors introduce TVP, a family of losses derived from EAL that bridges the gap between "coverage" (learning which tokens are likely) and "acceptance" (matching the target distribution exactly).
The core of TVP is a power parameter, $\beta \in [0, 1]$: $$D_\beta(p, q) = \frac{1}{\beta} \sum_v p(v) \left[ 1 - \min\left(1, \frac{q(v)}{p(v)}\right)^\beta \right]$$
- When $\beta = 1$: The loss is equivalent to Total Variation distance, focusing purely on acceptance.
- When $\beta \to 0$: The loss becomes a logarithmic surrogate that strongly penalizes undercovered tokens, providing a robust "coverage" signal.
The key innovation is Annealing $\beta$: Training starts with $\beta=0$ to quickly align the drafter with the target's high-probability tokens. As training progresses, $\beta$ is linearly increased to 1, shifting the objective toward the exact metric used in speculative sampling. This approach provides the strong gradients of KL divergence early on while ending with the theoretically optimal TV objective.
Datasets
- Training: The models were trained on Open-PerfectBlend, using "thinking-on" completions sampled from target models at temperature $T=1$.
- Evaluation: The authors tested on a suite of 9 benchmarks covering:
- Math: GSM8K, MATH, AIME25
- Code: MBPP, HumanEval, LiveCodeBench
- Chat: MT-Bench, Alpaca, Arena-Hard
Key Insights and Ideas
- TV is a Sleeping Giant: One surprising finding is that a well-tuned, plain TV baseline often matches or beats many complex, recently published objectives for fine-tuning. The authors argue that a strong TV baseline should be the standard for any new research in this area.
- The Vanishing Gradient Visualized: Figure 1 perfectly illustrates the problem. When the drafter ($q$) is much smaller than the target ($p$) for a specific token (like $y_3$), the TV gradient disappears. TVP at $\beta=0$ maintains a strong corrective signal.
- Coverage then Acceptance: The training curves in Figure 2 show that while TVP might start slower than KL-based methods, it eventually overtakes them. The initial "coverage" phase sets up a better global solution that the "acceptance" phase then refines.
Key Results
TVP consistently achieved the highest or tied-highest acceptance lengths across three different drafter architectures (DSpark, DFlash2, Eagle3) and three target models (Qwen3-8B, 14B, and 27B).
- Fine-tuning: TVP improved acceptance length by 9.8% over standard Cross-Entropy and 1.5% over plain TV.
- From Scratch: When training a drafter from random initialization, TVP outperformed TV by 4.2%.
- Speedups: On the Qwen3-8B DSpark model, TVP reached a 2.94x speedup over the target model alone, surpassing the released head and all other tested baselines (see Table 1).
Limitations and Future Work
- Fixed Budgets: The current annealing schedule assumes the training budget (number of steps/epochs) is known in advance. Developing adaptive schedules that don't require this knowledge is a potential area for future work.
- Greedy vs. Sampling: While TVP excels at $T=1$ (sampling), the paper notes that for greedy decoding ($T=0$), objectives with a KL or CE component still perform very well. Extending the TVP framework to better handle the $T=0$ regime is an open question.
- Advanced Verification: The authors showed TVP works well with block verification, but further exploration into tree-based verification or other non-linear verification schemes could yield even higher gains.
Summary updated