SpecSearch: Towards Hardware-Aware Efficient Speculative Decoding

Selin Yildirim, Ali Zeyrek, Deming ChenarXiv 2026 · arXiv

TL;DR

SpecSearch is a hardware-aware search engine that automatically optimizes speculative decoding configurations for Large Language Models (LLMs). By jointly tuning decoding strategies, draft lengths, and numerical precision based on specific GPU hardware and workload characteristics, it achieves up to a 2.01× throughput improvement over standard autoregressive decoding and 1.36× over baseline speculative decoding in vLLM.

What Problem Does This Paper Address?

Speculative decoding (SD) is a popular technique to speed up LLM inference. It uses a small, fast "drafter" model to propose tokens that a large "target" model then verifies in parallel. While SD can significantly reduce latency, its effectiveness isn't guaranteed.

The authors identify a major gap: the "optimal" configuration for SD is a moving target. It depends on the specific GPU (memory bandwidth vs. compute power), the workload (batch size and output length), and available memory. For instance, on an NVIDIA RTX 5090, SD might be 3.5× faster than standard autoregressive (AR) decoding at low concurrency, but actually slower (0.76×) at high concurrency because the system becomes compute-bound. Furthermore, drafter models consume precious GPU memory that could otherwise be used for the KV cache, potentially limiting the number of concurrent requests the system can handle.

Methods

SpecSearch tackles this multi-dimensional optimization problem through two primary components:

  1. Analytical Cost Model: This model characterizes the device's memory budget, bandwidth, and peak compute. It identifies which configurations (precision, draft length, etc.) are even feasible within memory limits and estimates the "critical batch size" ($B^*$) where the benefits of speculation begin to diminish due to compute contention.
  2. Policy Selection Engine: SpecSearch offers two modes to find the best configuration:
  3. Full Calibration: It runs offline measurements on the target GPU, guided by the cost model, to build a "policy table" that maps workloads to the best-performing settings.
  4. Learning-Based Predictor: To avoid time-consuming calibration on every new GPU, the authors trained a "Jev-based" predictor. This is a parameter-efficiently fine-tuned Qwen3-4B model (using LoRA) that can predict the optimal decoding configuration for a "held-out" (unseen) GPU in a zero-shot manner.
Overview of the SpecSearch workflow, from hardware inputs to the final policy table.

Datasets

The authors evaluated SpecSearch using the Qwen3-4B model as the target and DFlash as the drafter. They tested across a variety of standard LLM benchmarks, including:

  • GSM8K and MATH-500 (Reasoning)
  • MBPP and HumanEval (Coding)
  • MT-Bench (General dialogue)

Evaluation was performed on a diverse set of hardware: NVIDIA RTX 5090, H100, L40, and AMD MI300X.

Key Insights and Ideas

  • The Compute-Bound Trap: Speculative decoding is most effective when the system is memory-bound (common at low batch sizes). As batch size increases, the system shifts to being compute-bound, and the overhead of the drafter model can actually decrease total throughput.
  • Precision Matters: Switching the KV cache from fp16 to fp8 precision doubles the cache capacity. SpecSearch finds that this is a massive win for long-output workloads where memory capacity is the bottleneck, but offers negligible gains for short-output tasks.
  • Dynamic Draft Lengths: Rather than using a fixed draft length (e.g., always guessing 15 tokens), SpecSearch can dynamically adjust the draft length based on the effective batch size at each decoding step, maximizing efficiency as requests finish and new ones enter the batch.
Throughput comparison showing how SD advantage disappears at high concurrency.

Key Results

  • Throughput Gains: Across four different GPUs, SpecSearch with full calibration achieved an average of 2.01× the throughput of AR and 1.36× the throughput of standard SD in vLLM.
  • Predictor Accuracy: The zero-shot Jev-based predictor was remarkably effective, retaining 97.36% of the throughput achieved by the full-calibration mode, effectively eliminating the need for per-GPU profiling.
  • Memory Efficiency: The analytical cost model was highly accurate, predicting total GPU memory footprint within 0.1% to 0.2% on both AMD and NVIDIA hardware.
  • Comparison with State-of-the-Art: SpecSearch significantly outperformed MagicDec, largely because SpecSearch's hardware-aware model better manages the memory trade-offs required for high-concurrency serving.

Limitations and Future Work

The authors acknowledge a few areas for growth:

  • Model Scope: The current study focuses on a single target model (Qwen3-4B) and a diffusion-based drafter (DFlash). Future work could extend this to other architectures like EAGLE or Medusa.
  • Predictor Complexity: While the predictor is excellent at selecting static configurations, it does not yet predict the optimal dynamic draft-length mappings directly; these are currently derived from calibration data.
  • Moderate Concurrency Gaps: The predictor showed some performance gaps at moderate concurrency levels on specific high-end hardware like the H100, suggesting a need for even more diverse training data.

Summary updated

Abstract

Speculative decoding (SD) accelerates large language model (LLM) inference, but its optimal configuration depends jointly on workload characteristics, GPU hardware, and available memory. For example, on the RTX~5090, the state-of-the-art speculative drafter DFlash achieves $3.50\times$ the throughput of autoregressive (AR) decoding at low concurrency, but only $0.76\times$ at high concurrency with short outputs. We present SpecSearch, a hardware-aware search engine that jointly optimizes decoding strategy, draft length, and precision across workload--GPU pairs. Its analytical cost model captures memory and compute constraints to identify feasible configurations and narrow the search space for throughput optimization. Guided by this model, calibration measurements are collected from multiple GPUs. Using the measurements, a Jev-based predictor learns to select high-throughput decoding configurations on held-out GPUs without requiring deployment-GPU calibration. The predictor is parameter-efficiently trained on calibration data, while adapting a frozen Qwen3-4B using LoRA adapters and a linear correction layer. Across NVIDIA RTX~5090, H100, L40 and AMD MI300X GPUs, SpecSearch with full calibration achieves an average of $2.01\times$ the throughput of AR and $1.36\times$ the throughput of SD, both served by vLLM. The proposed Jev-based predictor closely matches this search performance with $1.30\times$ throughput improvement over SD with vLLM, while eliminating the need for held-out GPU calibration. Additionally, it retains $97.36\%$ of the throughput achieved by SpecSearch that utilizes full calibration.
BibTeX
@misc{yildirim2026specsearch,
  author = {Selin Yildirim and Ali Zeyrek and Deming Chen},
  title = {{SpecSearch: Towards Hardware-Aware Efficient Speculative Decoding}},
  year = {2026},
  eprint = {2611.22296},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.22296},
}
Figure