Asynchronous Speculative Decoding for Parallel LLM Inference

This direction is merged into another direction: speculative decoding. Its last version stays readable and citable here.

Version 1 of 161 papers90 conceptsfrozen 2026-10-04

Standard Large Language Model (LLM) inference is notoriously slow because it is autoregressive: to generate the 100th token, you must first finish generating the 99th. Speculative decoding was introduced to break this by using a small "draft" model to guess a sequence of tokens, which a large "target" model then verifies in a single parallel pass.

However, even speculative decoding has a "sequential bottleneck." In the traditional "draft-then-verify" cycle, the target model sits idle while the draft model generates its guesses, and the draft model sits idle while the target model performs verification. This lockstep dependency means the total time for one step is the sum of the drafting time and the verification time.

Recent research has converged on a solution: Asynchronous Speculative Decoding. By decoupling these two phases, systems can overlap drafting and verification, effectively "hiding" the latency of the draft model and maximizing hardware utilization.

The Sequential Bottleneck and the Drafting Tax

In vanilla speculative decoding, the system follows a strict serial loop. As Tianyu Liu and colleagues point out in PEARL, this leads to significant hardware underutilization. If a draft model takes 20ms to propose five tokens and the target model takes 80ms to verify them, the total cycle is 100ms. The draft model is the "tax" you pay to potentially skip future target model passes.

Asynchronous Speculative Decoding (ASD) eliminates this tax. In an asynchronous setup, while the target model is verifying the current batch of tokens, the draft model is already speculatively generating the next batch. As Nadav Timor demonstrates with Distributed Speculative Inference (DSI), this "speculation parallelism" makes the process non-blocking. If the drafting time is shorter than the verification time, the drafting latency is completely hidden, reducing the 100ms cycle in our example to just the 80ms required by the target model.

Decoupling via Pipelining and Buffers

To make this work, the system must treat the draft and target models as independent producers and consumers. Xuezhu Wang and the authors of Minions implement this using an intermediate results pool. The "minion" models (drafters) continuously pump token guesses into a buffer, and the large model pulls from this buffer whenever it is ready to verify.

This pipelined approach is pushed further in PipeSpec by Bradley McDanel, which introduces a hierarchical pipeline. If a downstream verifier finds a mistake, it sends a "rollback" signal upstream. Because the stages are asynchronous, the drafter might have already generated several tokens past the error; the system simply flushes the invalid portion of the buffer and restarts from the last verified token.

Leveraging Heterogeneous Hardware

A key insight of this research direction is that the draft and target models don't need to compete for the same compute resources. Nikhil Bhendawade and the Mirror-SD team show that you can run the target model on a primary GPU while the draft model runs on a secondary accelerator, like an NPU or even a CPU.

Similarly, Honglin Guo introduces DuoDecoding, which offloads drafting to the CPU. By using different hardware units, the "drafting tax" is not just hidden in time, but also physically separated in space, ensuring that the heavy-duty GPU is never waiting for a smaller processor to finish its work.

Temporal Phase Staggering in Batched Inference

When serving multiple requests at once (batching), the benefits of asynchrony become even more pronounced. Zhenwei Tang and the creators of MineDraft use a technique called temporal phase staggering.

In a standard batched system, all requests in a batch must wait for the slowest request to finish verification before the next drafting phase begins. MineDraft maintains two separate request batches and staggers their execution: while the target model verifies Batch A, the draft model generates tokens for Batch B. This ensures that the GPU's compute cores (used for verification) and the memory bandwidth (often the bottleneck for drafting) are both kept busy, leading to throughput gains of up to 75%.

Speculating on the Speculation

The most advanced form of this idea is Speculative Speculative Decoding (SSD), proposed by Tanishq Kumar and Tri Dao. In vanilla speculative decoding, the drafter must wait for the actual verification result to know where to start the next draft. SSD breaks this final dependency by having the draft model "guess" the outcome of the verification itself.

By predicting which tokens the target model will accept, the drafter can continue generating far into the future without waiting for a single confirmation. This transforms the inference process from a sequence of "stop-and-go" steps into a continuous flow of speculative generation and background verification.

Why It Works: The Intuition of Overlap

The fundamental reason Asynchronous Speculative Decoding works is latency masking.

  1. In Serial SD: Total Time = $Time(Draft) + Time(Verify)$.
  2. In Async SD: Total Time = $max(Time(Draft), Time(Verify))$.

As long as the draft model is faster than the target model's verification pass—which is almost always true—the drafting process becomes "free" in terms of wall-clock time. By treating LLM generation as a distributed systems problem rather than just a modeling problem, researchers like Chengquan Jiang (SwiftSpec) have been able to push inference speeds to hundreds of tokens per second, finally overcoming the sequential nature of the Transformer.

Papers in this version

Showing 60 of 61 papers.

History & citationVersion 1 · 2026-10-04

Merged into another direction · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_asynchronous_speculative_decoding_for_pa_43cfff27,
  title = {Asynchronous Speculative Decoding for Parallel LLM Inference},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/asynchronous-speculative-decoding-for-parallel-llm-inference/rdr_43cfff27d9964b0f80c96d6ae8a909e1},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_asynchronous_speculative_decoding_for_pa_43cfff27_v1,
  title = {Asynchronous Speculative Decoding for Parallel LLM Inference},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/asynchronous-speculative-decoding-for-parallel-llm-inference/rdr_43cfff27d9964b0f80c96d6ae8a909e1/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}