TL;DR
Hybrid Spark is a framework that dynamically selects the best speculative decoding drafter—either a neural model or a simple n-gram lookup—for every request at every step. By using a unified "slot price" to value verification compute, it achieves up to 1.62x throughput gains on repetitive tasks like code editing while maintaining robust performance on general chat.
What Problem Does This Paper Address?
Speculative decoding (SD) is a popular technique to speed up Large Language Model (LLM) inference. It works by using a "drafter" to guess a sequence of tokens, which the larger "target" model then verifies in a single pass. However, the effectiveness of SD is highly sensitive to content.
Neural drafters (like DSpark) are good at general text but usually propose short blocks. In contrast, model-free drafters (like n-gram lookups) can propose very long sequences for repetitive content—like code being edited—at almost zero cost. Currently, serving systems must choose one drafter for all traffic or split their hardware between them. Furthermore, existing "adaptive verification" systems, which adjust how many tokens to verify based on confidence, only support a single neural drafter because model-free drafters don't provide per-token confidence scores.
Methods
Hybrid Spark introduces a way to run both a model-based and a model-free drafter simultaneously on the GPU, choosing the better chain for each request without breaking the asynchronous scheduling of modern inference engines like vLLM.
- Calibrated Acceptance Estimates: The authors developed a logistic regression estimator to give n-gram matches a "confidence score." It uses simple features like match length ($n$), copy length ($m$), and "agreement" between multiple matches. This allows n-gram drafts to participate in budget allocation alongside neural drafts.
- The "Slot Price" ($\lambda$): To compare different drafters, the authors treat verification slots as a resource with an opportunity cost. They define a price $\lambda$ representing the "goodput" (expected tokens per second) a slot would earn elsewhere in the batch. A draft token is only worth verifying if its survival probability exceeds this price.
- Chain Selection Rule: At every step, the system calculates the net value ($V$) of both candidate chains: $V = \sum_k \max(S_k - \lambda, 0)$, where $S_k$ is the probability the $k$-th token is accepted. It picks the chain with the higher value.
- The "Contains Rule": If an n-gram match is long and happens to start with the same tokens the neural drafter predicted, Hybrid Spark merges them, using the neural drafter's superior accuracy for the first few tokens and the n-gram's length for the tail.
Datasets
The authors evaluated Hybrid Spark across a diverse set of models and workloads:
- Models: Qwen3-4B/8B, Gemma 4 12B, Llama-3.1-8B, and North-Mini-Code-1.0.
- General Benchmarks: SPEED-Bench (used for fitting the estimator and general evaluation) and MT-Bench (for chat).
- Code Editing Benchmarks: NextCoder, EditBench, Blazedit 5k/10k, and CanItEdit. These are crucial because they contain the repetitive "copy-paste" patterns where n-gram drafters shine.
Key Insights and Ideas
- Opposite Acceptance Profiles: As shown in Figure 2, neural drafters are most accurate at the start of a draft, while n-gram drafters are weakest at the first token but become extremely reliable once a "copy" sequence is established.
- Unified Scaling: By converting "confidence" into "expected goodput minus price," the authors created a common currency that allows a serving system to decide whether to verify a short, high-confidence neural draft or a long, medium-confidence n-gram draft.
- Zero-Synchronization Design: All token-dependent decisions (selection and allocation) happen entirely on the GPU. The CPU host only updates the "price" and "budget" every few steps, which prevents the system from stalling due to host-device communication.
Key Results
- Code Editing Speedups: On the NextCoder workload (Qwen3-4B), Hybrid Spark achieved 1.10–1.62x the throughput of DSpark at batch size 4. Even at a large batch size of 128, it maintained a 1.04–1.27x gain.
- Robustness: On general chat (MT-Bench), where n-gram matches are rare, the system stayed within 0.97–1.01x of DSpark, proving it doesn't hurt performance when the model-free drafter isn't useful.
- Content Sensitivity: The gains are highest when the output is nearly identical to the input (low edit distance). As the model performs heavier rewrites, the system naturally shifts its preference back to the neural drafter.
Limitations and Future Work
- Probabilistic Drafting: The value-based selection rule is not strictly "lossless" (it might slightly bias the output distribution) when using probabilistic sampling ($T > 0$). The authors propose simpler "extension" policies to maintain losslessness, but these are slightly less performant.
- Stale Budgeting: Because the budget is sized based on data from two steps ago, the system can sometimes under- or over-allocate slots when n-gram matches appear or disappear suddenly. While the impact is small at high batch sizes, it's a measurable overhead at small batch sizes.
- Future Drafters: The framework is designed to be modular. A natural next step is integrating history-based drafters for Reinforcement Learning (RL) rollout generation, where prompts often repeat across training epochs.
Summary updated