TVP: TV is Almost All You Need for Speculative Decoding

Aaryam SharmaarXiv 2026 · arXiv

TL;DR

TVP (Total Variation Power) is a new training objective for speculative decoding that solves the "vanishing gradient" problem inherent in standard Total Variation (TV) distance. By annealing from a coverage-focused loss to an acceptance-focused one, TVP consistently outperforms cross-entropy and complex specialized objectives, delivering up to a 12.9% improvement in acceptance length across various LLM architectures.

What Problem Does This Paper Address?

Speculative decoding is a popular technique to speed up Large Language Model (LLM) inference. It uses a small, fast "drafter" model to propose a sequence of tokens, which a large "target" model then verifies in parallel. The efficiency of this process hinges on the Expected Acceptance Length (EAL)—the average number of tokens the target model accepts from the drafter's proposal.

Theoretically, to maximize EAL, the drafter should minimize the Total Variation (TV) distance between its distribution and the target's. However, TV has a major flaw: its gradient vanishes when the drafter puts very little probability on a token the target model likes (an "undercovered" token). This makes it nearly impossible to train a drafter from scratch using pure TV.

On the other hand, directly optimizing for EAL leads to "prefix attenuation." Because a token can only be accepted if all preceding tokens were also accepted, the training signal for later positions in a draft becomes incredibly weak. Most researchers fall back on Cross-Entropy (CE) or KL divergence, which provide better gradients but don't directly optimize for the acceptance mechanism used during inference.

Methods

The authors introduce TVP, a family of losses derived from EAL that bridges the gap between "coverage" (learning which tokens are likely) and "acceptance" (matching the target distribution exactly).

The core of TVP is a power parameter, $\beta \in [0, 1]$: $$D_\beta(p, q) = \frac{1}{\beta} \sum_v p(v) \left[ 1 - \min\left(1, \frac{q(v)}{p(v)}\right)^\beta \right]$$

  • When $\beta = 1$: The loss is equivalent to Total Variation distance, focusing purely on acceptance.
  • When $\beta \to 0$: The loss becomes a logarithmic surrogate that strongly penalizes undercovered tokens, providing a robust "coverage" signal.

The key innovation is Annealing $\beta$: Training starts with $\beta=0$ to quickly align the drafter with the target's high-probability tokens. As training progresses, $\beta$ is linearly increased to 1, shifting the objective toward the exact metric used in speculative sampling. This approach provides the strong gradients of KL divergence early on while ending with the theoretically optimal TV objective.

Datasets

  • Training: The models were trained on Open-PerfectBlend, using "thinking-on" completions sampled from target models at temperature $T=1$.
  • Evaluation: The authors tested on a suite of 9 benchmarks covering:
  • Math: GSM8K, MATH, AIME25
    • Code: MBPP, HumanEval, LiveCodeBench
    • Chat: MT-Bench, Alpaca, Arena-Hard

Key Insights and Ideas

  • TV is a Sleeping Giant: One surprising finding is that a well-tuned, plain TV baseline often matches or beats many complex, recently published objectives for fine-tuning. The authors argue that a strong TV baseline should be the standard for any new research in this area.
  • The Vanishing Gradient Visualized: Figure 1 perfectly illustrates the problem. When the drafter ($q$) is much smaller than the target ($p$) for a specific token (like $y_3$), the TV gradient disappears. TVP at $\beta=0$ maintains a strong corrective signal.
Figure 1. Target and draft probabilities (top) and score-gradient weights divided by wi (bottom) on one target-sampled continu- ation. TV barely corrects the undercovered y3, and EAL also attenuates later positions, while TVP at β = 0 retains correction throughout the window.
  • Coverage then Acceptance: The training curves in Figure 2 show that while TVP might start slower than KL-based methods, it eventually overtakes them. The initial "coverage" phase sets up a better global solution that the "acceptance" phase then refines.
Training curves showing TVP (blue) overtaking other objectives as $\beta$ anneals toward 1.

Key Results

TVP consistently achieved the highest or tied-highest acceptance lengths across three different drafter architectures (DSpark, DFlash2, Eagle3) and three target models (Qwen3-8B, 14B, and 27B).

  • Fine-tuning: TVP improved acceptance length by 9.8% over standard Cross-Entropy and 1.5% over plain TV.
  • From Scratch: When training a drafter from random initialization, TVP outperformed TV by 4.2%.
  • Speedups: On the Qwen3-8B DSpark model, TVP reached a 2.94x speedup over the target model alone, surpassing the released head and all other tested baselines (see Table 1).
Table 1. Qwen3-8B DSpark, fine-tuned for one epoch. Speedup is over the target alone at one request, and ± is the 95% CI.

Limitations and Future Work

  • Fixed Budgets: The current annealing schedule assumes the training budget (number of steps/epochs) is known in advance. Developing adaptive schedules that don't require this knowledge is a potential area for future work.
  • Greedy vs. Sampling: While TVP excels at $T=1$ (sampling), the paper notes that for greedy decoding ($T=0$), objectives with a KL or CE component still perform very well. Extending the TVP framework to better handle the $T=0$ regime is an open question.
  • Advanced Verification: The authors showed TVP works well with block verification, but further exploration into tree-based verification or other non-linear verification schemes could yield even higher gains.

Summary updated

Abstract

Speculative decoding accelerates LLM inference with a drafter whose proposed tokens are verified by the target model in parallel. Under lossless speculative sampling, a draft token is accepted with probability one minus the total variation (TV) distance between the target and draft distributions, yet drafters are often trained with cross-entropy (CE) or KL. We first show that under matched and well-tuned training, TV matches or surpasses several published draft-training objectives when fine-tuning, and a fixed TV and KL mixture does the same when training from scratch. We identify two remaining problems, TV's vanishing gradients on undercovered tokens and the prefix products that starve later positions in acceptance-length objectives. To address both, we derive TVP, a loss family based on expected acceptance length, whose power parameter interpolates between coverage and acceptance oriented objectives. Annealing the power from zero to one gives the highest or tied-highest acceptance length across three drafter architectures and three target models, including longer drafts and block verification. TVP improves acceptance length over CE by 9.8% when fine-tuning, and over the public DFlash2 training recipes by 6.2 to 12.9%.
BibTeX
@misc{sharma2026tvp,
  author = {Aaryam Sharma},
  title = {{TVP: TV is Almost All You Need for Speculative Decoding}},
  year = {2026},
  eprint = {2611.00588},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.00588},
}
Figure