Hybrid Spark: Adaptive Speculative Decoding That Lets the Content Choose the Drafter

Ekagra Ranjan, Bharat VenkitesharXiv 2026 · arXiv

TL;DR

Hybrid Spark is a framework that dynamically selects the best speculative decoding drafter—either a neural model or a simple n-gram lookup—for every request at every step. By using a unified "slot price" to value verification compute, it achieves up to 1.62x throughput gains on repetitive tasks like code editing while maintaining robust performance on general chat.

What Problem Does This Paper Address?

Speculative decoding (SD) is a popular technique to speed up Large Language Model (LLM) inference. It works by using a "drafter" to guess a sequence of tokens, which the larger "target" model then verifies in a single pass. However, the effectiveness of SD is highly sensitive to content.

Neural drafters (like DSpark) are good at general text but usually propose short blocks. In contrast, model-free drafters (like n-gram lookups) can propose very long sequences for repetitive content—like code being edited—at almost zero cost. Currently, serving systems must choose one drafter for all traffic or split their hardware between them. Furthermore, existing "adaptive verification" systems, which adjust how many tokens to verify based on confidence, only support a single neural drafter because model-free drafters don't provide per-token confidence scores.

Methods

Hybrid Spark introduces a way to run both a model-based and a model-free drafter simultaneously on the GPU, choosing the better chain for each request without breaking the asynchronous scheduling of modern inference engines like vLLM.

  1. Calibrated Acceptance Estimates: The authors developed a logistic regression estimator to give n-gram matches a "confidence score." It uses simple features like match length ($n$), copy length ($m$), and "agreement" between multiple matches. This allows n-gram drafts to participate in budget allocation alongside neural drafts.
  2. The "Slot Price" ($\lambda$): To compare different drafters, the authors treat verification slots as a resource with an opportunity cost. They define a price $\lambda$ representing the "goodput" (expected tokens per second) a slot would earn elsewhere in the batch. A draft token is only worth verifying if its survival probability exceeds this price.
  3. Chain Selection Rule: At every step, the system calculates the net value ($V$) of both candidate chains: $V = \sum_k \max(S_k - \lambda, 0)$, where $S_k$ is the probability the $k$-th token is accepted. It picks the chain with the higher value.
  4. The "Contains Rule": If an n-gram match is long and happens to start with the same tokens the neural drafter predicted, Hybrid Spark merges them, using the neural drafter's superior accuracy for the first few tokens and the n-gram's length for the tail.
Figure 3. Hybrid Spark overview, instantiated with DSpark and an n-gram drafter. Every per-step decision runs on the GPU (blue), while the CPU host (orange) sizes the budget and sets the slot price λ from survivals produced two steps prior. Solid arrows stay on the GPU, and dashed arrows carry host scalars or asynchronous copies.

Datasets

The authors evaluated Hybrid Spark across a diverse set of models and workloads:

  • Models: Qwen3-4B/8B, Gemma 4 12B, Llama-3.1-8B, and North-Mini-Code-1.0.
  • General Benchmarks: SPEED-Bench (used for fitting the estimator and general evaluation) and MT-Bench (for chat).
  • Code Editing Benchmarks: NextCoder, EditBench, Blazedit 5k/10k, and CanItEdit. These are crucial because they contain the repetitive "copy-paste" patterns where n-gram drafters shine.

Key Insights and Ideas

  • Opposite Acceptance Profiles: As shown in Figure 2, neural drafters are most accurate at the start of a draft, while n-gram drafters are weakest at the first token but become extremely reliable once a "copy" sequence is established.
  • Unified Scaling: By converting "confidence" into "expected goodput minus price," the authors created a common currency that allows a serving system to decide whether to verify a short, high-confidence neural draft or a long, medium-confidence n-gram draft.
  • Zero-Synchronization Design: All token-dependent decisions (selection and allocation) happen entirely on the GPU. The CPU host only updates the "price" and "budget" every few steps, which prevents the system from stalling due to host-device communication.
Figure 2. The two drafters have opposite acceptance profiles. Conditional acceptance αi by draft position (Qwen3-4B; higher is better). Given a match, the n-gram drafter’s first token is its weakest and its deep tokens are accepted more often than any of DSpark’s. The n-gram curve counts only steps with a match.

Key Results

  • Code Editing Speedups: On the NextCoder workload (Qwen3-4B), Hybrid Spark achieved 1.10–1.62x the throughput of DSpark at batch size 4. Even at a large batch size of 128, it maintained a 1.04–1.27x gain.
  • Robustness: On general chat (MT-Bench), where n-gram matches are rare, the system stayed within 0.97–1.01x of DSpark, proving it doesn't hurt performance when the model-free drafter isn't useful.
  • Content Sensitivity: The gains are highest when the output is nearly identical to the input (low edit distance). As the model performs heavier rewrites, the system naturally shifts its preference back to the neural drafter.
Figure 1. The best drafter depends on content, and Hybrid Spark tracks the better one. Output-throughput speedup over vanilla decoding on NextCoder (Qwen3-4B), by edit distance between the input program and the model’s output (higher is better). The n-gram drafter outperforms DSpark only on near-copies, and verifying all of its drafts (n-gram) at BS 128 is slower than vanilla decoding from 35% edit distance up. N-gram + AV gives it adaptive verification with our estimator. Table 11 in the appendix lists the values.

Limitations and Future Work

  • Probabilistic Drafting: The value-based selection rule is not strictly "lossless" (it might slightly bias the output distribution) when using probabilistic sampling ($T > 0$). The authors propose simpler "extension" policies to maintain losslessness, but these are slightly less performant.
  • Stale Budgeting: Because the budget is sized based on data from two steps ago, the system can sometimes under- or over-allocate slots when n-gram matches appear or disappear suddenly. While the impact is small at high batch sizes, it's a measurable overhead at small batch sizes.
  • Future Drafters: The framework is designed to be modular. A natural next step is integrating history-based drafters for Reinforcement Learning (RL) rollout generation, where prompts often repeat across training epochs.

Summary updated

Abstract

Speculative decoding (SD) speeds up large language model (LLM) serving, but its gain depends on the workload's \emph{shape}, such as batch size and context length, and on its \emph{content}. A model-based drafter proposes a short block on any content, while a model-free drafter such as n-gram lookup proposes long copies at almost no cost when the output repeats its input, as in code editing. Adaptive verification spends a batch-wide verification budget on the draft tokens most likely to be accepted, but it supports only one model-based drafter. Model-free drafters provide no per-token confidence, so they cannot take part in budget allocation, and deployments must either fix one drafter or split their GPUs between drafters. We present \sys, a framework that runs a model-based and a model-free drafter on every step and gives each request the better of their chains, within adaptive verification and entirely on the GPU, which keeps asynchronous scheduling intact. The key idea is a price per verified draft token: the goodput that slot would earn elsewhere in the batch. Once the model-free drafter's tokens carry calibrated acceptance estimates, fit once on general text, this price puts both drafters' chains on one scale. Instantiated with \dspark and an n-gram drafter in vLLM, \sys achieves 1.10--1.62\x the output throughput of \dspark on code editing at batch size~4 and 1.04--1.27\x at batch size~128, while staying at 0.97--1.01\x of it on chat at temperature~0. A single deployment can therefore serve both workloads.
BibTeX
@misc{ranjan2026hybrid,
  author = {Ekagra Ranjan and Bharat Venkitesh},
  title = {{Hybrid Spark: Adaptive Speculative Decoding That Lets the Content Choose the Drafter}},
  year = {2026},
  eprint = {2611.05611},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.05611},
}
Figure