The Latency Bottleneck of Multi Stage Decoding

16 concepts13 papers1 proposal

Autoregressive language models generate text token-by-token by maximizing local conditional probabilities. While computationally straightforward, this token-level optimization often fails to align with global, sentence-level quality. To bridge this gap, state-of-the-art systems frequently deploy multi-stage decoding pipelines. Instead of relying on a single decoding pass, these architectures generate a broad pool of candidate hypotheses and use secondary models to filter, score, or rerank them.

While multi-stage decoding consistently wins competitive benchmarks, it introduces a severe inference latency bottleneck. The computational overhead of running multiple generation and evaluation passes makes these high-performing systems notoriously difficult to deploy in real-time production environments.


The Mechanics of the Multi-Pass Tax

To understand why multi-stage decoding explodes latency, we must look at the two distinct phases required by these pipelines: candidate generation and secondary evaluation.

[Input Prompt] ──> [Base Model Sampling] ──> [Candidate Pool (N Hypotheses)] ──> [Secondary Reranker / MBR] ──> [Best Output]

1. Candidate Generation Overhead

Standard greedy search tracks a single path. Multi-stage decoding, by contrast, requires generating dozens or even hundreds of diverse alternative sequences to ensure high-quality candidates are captured. For example, Hiroyuki Deguchi et al. demonstrated a system in NAIST-NICT WMT'23 General MT Task Submission that generated up to 900 diverse translation candidates using varied architectures and decoding strategies. Scaling the candidate pool to this magnitude consumes massive computational resources before evaluation even begins.

2. Secondary Evaluation Overhead

Once candidates are generated, they must be processed by a secondary scoring mechanism. Rather than relying on the base model's internal token probabilities, "quality-aware decoding" frameworks—such as the one proposed by Patrick Fernandes et al. in Quality-Aware Decoding for Neural Machine Translation—evaluate candidates using learnable, high-capacity metrics like COMET or BLEURT. Similarly, Pedram Rooshenas et al. introduced Energy-Based Reranking: Improving Neural Machine Translation Using Energy-Based Models, which routes candidates through a heavy, BERT-powered energy model.

As José P. Pombal et al. noted in Tower v2: Unbabel-IST 2024 Submission for the General MT Shared Task, executing these multiple forward passes through external neural evaluation models introduces a prohibitive latency penalty.

3. The Quadratic Cost of Consensus

The latency bottleneck becomes even more pronounced when pipelines incorporate consensus-based selection techniques like Minimum Bayes Risk (MBR) decoding. MBR finds the candidate that minimizes expected risk by performing pairwise quality comparisons across the entire pool. If a system generates $N$ candidates, MBR requires $O(N^2)$ evaluations. As highlighted by Keito Kudo et al. in SKIM at WMT 2023 General Translation Task, combining Quality Estimation (QE) models with MBR decoding achieves remarkable accuracy but creates an exceptionally slow pipeline that is impractical for time-sensitive production environments.


A Pervasive Bottleneck Across Tasks

This tension between accuracy and inference speed is a recurring theme across diverse natural language processing domains:


Baking Reranking into a Single Pass

To circumvent the multi-pass tax, recent research has shifted toward a new paradigm: distilling the intelligence of multi-stage pipelines directly into the base model's weights. The goal is to achieve the quality gains of reranking or MBR decoding within a single, fast inference pass.

Multi-Stage (Slow):  [Base Model] ──> [N Candidates] ──> [External Reranker] ──> [Output]
Single-Pass (Fast):  [Aligned/Calibrated Model] ───────────────────────────────> [Output]

Direct Preference Optimization (DPO)

Instead of running expensive MBR selection at runtime, the model can be trained to inherently prefer high-consensus outputs. In Direct Preference Optimization for Neural Machine Translation with Minimum Bayes Risk Decoding, Guangyu Yang et al. utilized DPO to "bake" MBR benefits directly into the model. By training the network on pairs of high- and low-risk translations generated offline, the authors achieved MBR-level translation quality using standard, single-pass decoding.

Confidence Calibration

Another approach addresses the root cause of why reranking is necessary: poorly calibrated token probabilities. If a model's internal confidence does not correlate with actual output quality, it will select suboptimal tokens during standard decoding. Di Wu et al. tackled this in Calibrating Translation Decoding with Quality Estimation on LLMs by optimizing the Pearson correlation between a model's internal token confidence and external quality metrics. The resulting calibrated model can self-evaluate effectively, allowing efficient beam search to yield the same accuracy as an expensive external reranking pipeline.

Dynamic Hesitation

Rather than expanding the search space uniformly for every token or sentence, models can allocate computational budgets dynamically based on difficulty. Romain Storaï introduced a training-free, plug-and-play method called HARP in HARP: Hesitation-Aware Reframing in Transformer Inference Pass. HARP monitors token-level uncertainty during the inference pass. When the model "hesitates" on a difficult token, the system dynamically reframes the input using embedding dropout, allowing the model to rethink its choice on the fly. This targeted intervention provides performance gains while remaining significantly faster than traditional multi-candidate search methods.

Key Foundational Work

NVIDIA NeMo's Neural Machine Translation Systems for English-German and English-Russian News and Biomedical Tasks at WMT21 (2021) — contributed 2 concept elements to this direction.
Energy-Based Reranking: Improving Neural Machine Translation Using Energy-Based Models (2020) — contributed 1 concept elements to this direction.
Searchable Hidden Intermediates for End-to-End Models of Decomposable Sequence Tasks (2021) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_multi_stage_decoding_latency_19cc84e8,
  title = {multi-stage decoding latency},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/multi-stage-decoding-latency/rdr_19cc84e8f02349db87d9de997450d408},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-21},
}

This exact version (version 1, frozen text):

@misc{lacuna_multi_stage_decoding_latency_19cc84e8_v1,
  title = {multi-stage decoding latency},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/multi-stage-decoding-latency/rdr_19cc84e8f02349db87d9de997450d408/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-21},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake