VIA-SD: Verification via Intra-Model Routing for Speculative Decoding

Yuchen Xian, Yang He, Yunqiu Xu, Yi Yang

The Bottleneck of Autoregressive Decoding

Large Language Models (LLMs) are notoriously slow because they generate text autoregressively—one token at a time. Speculative Decoding (SD) was introduced to mitigate this bottleneck by pairing a tiny, fast "drafter" model with a massive "verifier" model. The drafter guesses a sequence of tokens, and the verifier checks them all in parallel, significantly speeding up inference.

However, current SD frameworks operate on a strict binary: either the verifier perfectly agrees with the drafter, or it rejects the token and recomputes everything from that point forward. Researchers have identified a major inefficiency in this approach: the "middle zone." Many tokens are rejected because the drafter was only slightly off, yet the system forces a massive (e.g., 70B parameter) model to make the correction. Using a full-scale model for minor corrections is a massive computational waste.

VIA-SD: A Multi-Tiered Approach

To address this, the authors propose VIA-SD (Verification via Intra-Model Routing for Speculative Decoding). VIA-SD evolves speculative decoding from a binary accept/reject system into a hierarchical, three-tier framework:

  1. Drafter ($p$): Rapidly proposes candidate tokens.
  2. Slim-Verifier ($q'$): An intermediate model that handles medium-confidence "middle zone" tokens.
  3. Full Verifier ($q$): The original large model, reserved strictly for the most uncertain cases.

Key Innovations

  • Intra-Model Routing: Instead of training a separate medium-sized model—which often suffers from a "distribution mismatch" with the large model—VIA-SD creates the slim-verifier by strategically skipping layers of the full verifier. Because it shares the same embeddings and output head, the slim-verifier speaks the exact "same language" as the full model. The paper reveals that a layer-skipped 70B model is actually more efficient at verification than an independent 13B model.
  • Dynamic Intra-Model Routing (DIMR): To determine exactly which layers to skip, the team uses an offline search combining Bayesian optimization and random search. This quickly identifies the most efficient sub-structure for a specific model pair, taking only 18–68 minutes.
  • Hierarchical Gating: The system uses two confidence thresholds ($\alpha_1, \alpha_2$). High-confidence tokens are accepted immediately; medium-confidence tokens are routed to the slim-verifier; and low-confidence tokens escalate to the full model.
  • Information-Theoretic Design: Moving away from standard Total Variation (TV) distance, the authors use KL-divergence geometry to mathematically prove that a multi-step verification path ($p \to q' \to q$) is computationally more efficient than a direct jump ($p \to q$).

Performance and Results

The authors evaluated VIA-SD across diverse tasks (summarization, translation, reasoning, QA) and model families (T5, Gemma2, LLaMA2, Qwen).

  • Significant Speedups: VIA-SD delivers 10–20% speedups over state-of-the-art speculative decoding baselines and 2.5–3× acceleration compared to standard greedy decoding.
  • Fewer Rejections: It reduces token rejection rates by 0.10–0.22. A significant portion of rejected tokens can be correctly verified by a slim-verifier with 45% fewer layers.
  • Plug-and-Play Compatibility: VIA-SD requires no retraining of the base models. It can be layered seamlessly on top of existing advanced SD frameworks (like EAGLE or PEARL) to squeeze out even more performance.
  • Quality Preservation: Despite using confidence thresholds, task-level accuracy (e.g., ROUGE or BLEU scores) is maintained or slightly improved, as the hierarchical gating acts as a stabilizer.

Limitations and Future Directions

While VIA-SD presents a highly efficient paradigm for LLM inference, it relies on static routing; the slim-verifier's structure is fixed after the offline search. The authors suggest that developing learnable gates—which could dynamically skip layers based on the specific prompt—is a promising future direction.

Additionally, the system relies on two confidence thresholds ($\alpha_1, \alpha_2$). While default settings work well generally, optimal values can vary by task. Finally, real-world wall-clock speedups will still depend on hardware-specific factors, such as KV-cache management and kernel fusion, which vary across deployment environments.

Summary updated

Abstract

Speculative decoding (SD) addresses the high inference costs of LLMs by having lightweight drafters generate candidates for large verifiers to validate in parallel. Existing draft-verify methods use binary decisions: accept or fully recompute. Yet we find that many rejected tokens can be verified correctly by a slim submodel derived from the full verifier via intra-model routing, instead of the full verifier. This motivates our slim-verifier to handle tokens requiring moderate verification resources, reducing expensive large-model calls. We propose **V**erification via **I**ntr**a**-Model Routing for **S**peculative **D**ecoding (VIA-SD), a multi-tier framework using a routed slim-verifier. Draft tokens are processed hierarchically: direct acceptance for high-confidence cases, slim-verifier regeneration for medium-confidence cases, and full-model verification for uncertain cases. Across four representative tasks and multiple model families, VIA-SD reduces rejection rates by 0.10–0.22 and delivers 10–20% speedups over strong SD baselines, while achieving 2.5–3× acceleration over non-drafting decoding. Moreover, VIA-SD is compatible with existing SD frameworks without modifying their training procedures. Our results suggest multi-tier SD as a general paradigm for scalable and efficient LLM inference. Project page: https://zju-xyc.github.io/VIA-SD-Project-Page/
BibTeX
@inproceedings{xian2026via,
  author = {Yuchen Xian and Yang He and Yunqiu Xu and Yi Yang},
  title = {{VIA-SD: Verification via Intra-Model Routing for Speculative Decoding}},
  booktitle = {International Conference on Machine Learning},
  year = {2026},
  url = {https://openreview.net/forum?id=sGCDBHIr9u},
}
Figure