PipeSpec: Breaking Stage Dependencies in Hierarchical LLM Decoding

Bradley McDanel, Sai Qian Zhang, Yunhai Hu, Zining Liu

TL;DR

Speculative decoding accelerates large language model (LLM) inference but suffers from hardware inefficiency due to its synchronous "draft-then-verify" loop. PipeSpec solves this bottleneck by introducing an asynchronous, hierarchical pipeline of multiple draft models, eliminating idle time and achieving up to a 2.54× inference speedup.

What Problem Does This Paper Address?

Autoregressive decoding generates tokens one by one, making LLM inference heavily bound by memory bandwidth. Standard speculative decoding mitigates this by using a small, fast "draft" model to guess a sequence of tokens, which a large "target" model then verifies in parallel.

However, traditional speculative decoding operates in strict lockstep: the draft model idles while the target model verifies the batch, and the target model idles while the draft model generates the next sequence. This synchronous dependency leaves hardware underutilized and caps potential throughput gains.

Methods

The authors introduce PipeSpec, a framework that breaks these sequential stage dependencies using two core concepts:

  1. Asynchronous Pipelining: Instead of alternating between drafting and verifying, models run concurrently in a producer-consumer relationship. Draft models continuously generate tokens optimistically, assuming they will be accepted downstream.
  2. Hierarchical Speculation: PipeSpec generalizes speculative decoding to a pipeline of $k$ models of increasing size (e.g., $M_0 \rightarrow M_1 \rightarrow \dots \rightarrow M_k$). Each intermediate model acts as both a lightweight verifier for the preceding smaller model and a higher-quality drafter for the succeeding larger model.

When a downstream model rejects a token prediction due to a mismatch, it triggers a rollback cascade, signaling all earlier stages to discard the invalid predictions and resume generation from the last valid token.

Comparison of traditional autoregressive decoding, standard speculative decoding, and the hierarchical PipeSpec framework.

Datasets

The framework was evaluated on standard benchmarks across two primary tasks:

  • Text Summarization: CNN/DM and XSUM datasets.
  • Code Generation: The HumanEval benchmark.

Evaluations utilized the LLaMA-2 (7B, 13B) and LLaMA-3/3.1 (1B, 8B, 70B) model families.

Key Insights and Ideas

  • Breaking Lockstep is the Primary Driver: The main performance gain in PipeSpec comes from eliminating synchronization overhead rather than just improving draft quality. Keeping all GPUs continuously active prevents the alternating idle periods native to standard speculative decoding.
  • Progressive Token Refinement: In a three-model setup (e.g., 1B $\rightarrow$ 8B $\rightarrow$ 70B), the intermediate 8B model filters out low-quality drafts from the 1B model before they ever reach the computationally expensive 70B verification stage.
  • Exploiting the Long Tail: Unlike standard speculative decoding's rigid, fixed lookahead windows, PipeSpec allows natural token acceptance patterns to emerge. When models agree on "easy" token sequences, the pipeline can process very large token batches without artificial interruption.

Key Results

  • Significant Speedups: On the HumanEval benchmark with LLaMA 3.1-70B, a three-model PipeSpec pipeline ({1B, 8B, 70B}) achieved a 2.54× speedup over the autoregressive baseline, outperforming standard speculative decoding (1.37×).
  • Depth Correlates with Efficiency: Adding intermediate models consistently improved throughput. For instance, on XSum (LLaMA2-13B), a three-model setup achieved a 2.00× speedup compared to 1.64× for a two-model setup.
  • Better Hardware and Energy Efficiency: PipeSpec increased average GPU utilization from 23.0% (standard speculative decoding) to 39.7%. Despite higher sustained power draw, this continuous utilization reduced the total energy per token from 16.5J (autoregressive) to 5.8J due to the massive reduction in total inference time.
Figure 5: GPU utilization over time showing autore- gressive (70B model split across 2 GPUs), speculative decoding ({1B,8B,70B}), and PipeSpec ({1B,8B,70B}). PipeSpec achieves higher average utilization (39.7%) by eliminating idle periods between draft and verification.

Limitations and Future Work

  • Static Configuration: PipeSpec currently relies on a fixed selection of models and predetermined pipeline depths. It cannot dynamically adjust its architecture based on varying task complexity or real-time prediction patterns.
  • Rollback Overhead: If the draft model's predictions are highly inaccurate for a specific task, frequent mispredictions trigger expensive rollback cascades across multiple stages, degrading performance.
  • Memory Constraints: The framework assumes all models can fit into available GPU memory. While larger models can be split across devices, very deep pipelines with many intermediate models might hit strict hardware scaling limits.

Summary updated

Abstract

Speculative decoding accelerates large language model inference by using smaller draft models to generate candidate tokens for parallel verification. However, current approaches are limited by sequential stage dependencies that prevent full hardware utilization. We present PipeSpec, a framework that generalizes speculative decoding to $k$ models arranged in a hierarchical pipeline, enabling asynchronous execution with lightweight coordination for prediction verification and rollback. Our analytical model characterizes token generation rates across pipeline stages and proves guaranteed throughput improvements over traditional decoding for any non-zero acceptance rate. We further derive closed-form expressions for steady-state verification probabilities that explain the empirical benefits of pipeline depth. Experimental results show that PipeSpec achieves up to 2.54$\times$ speedup while outperforming state-of-the-art methods. We validate PipeSpec across text summarization and code generation tasks using LLaMA 2 and 3 models, demonstrating that pipeline efficiency increases with model depth, providing a scalable approach to accelerating LLM inference on multi-device systems.
BibTeX
@inproceedings{mcdanel2025pipespec,
  author = {Bradley McDanel and Sai Qian Zhang and Yunhai Hu and Zining Liu},
  title = {{PipeSpec: Breaking Stage Dependencies in Hierarchical LLM Decoding}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2025},
}
Figure