Autoregressive language models generate text token-by-token by maximizing local conditional probabilities. While computationally straightforward, this token-level optimization often fails to align with global, sentence-level quality. To bridge this gap, state-of-the-art systems frequently deploy multi-stage decoding pipelines. Instead of relying on a single decoding pass, these architectures generate a broad pool of candidate hypotheses and use secondary models to filter, score, or rerank them.
While multi-stage decoding consistently wins competitive benchmarks, it introduces a severe inference latency bottleneck. The computational overhead of running multiple generation and evaluation passes makes these high-performing systems notoriously difficult to deploy in real-time production environments.
The Mechanics of the Multi-Pass Tax
To understand why multi-stage decoding explodes latency, we must look at the two distinct phases required by these pipelines: candidate generation and secondary evaluation.
[Input Prompt] ──> [Base Model Sampling] ──> [Candidate Pool (N Hypotheses)] ──> [Secondary Reranker / MBR] ──> [Best Output]
1. Candidate Generation Overhead
Standard greedy search tracks a single path. Multi-stage decoding, by contrast, requires generating dozens or even hundreds of diverse alternative sequences to ensure high-quality candidates are captured. For example, Hiroyuki Deguchi et al. demonstrated a system in NAIST-NICT WMT'23 General MT Task Submission that generated up to 900 diverse translation candidates using varied architectures and decoding strategies. Scaling the candidate pool to this magnitude consumes massive computational resources before evaluation even begins.
2. Secondary Evaluation Overhead
Once candidates are generated, they must be processed by a secondary scoring mechanism. Rather than relying on the base model's internal token probabilities, "quality-aware decoding" frameworks—such as the one proposed by Patrick Fernandes et al. in Quality-Aware Decoding for Neural Machine Translation—evaluate candidates using learnable, high-capacity metrics like COMET or BLEURT. Similarly, Pedram Rooshenas et al. introduced Energy-Based Reranking: Improving Neural Machine Translation Using Energy-Based Models, which routes candidates through a heavy, BERT-powered energy model.
As José P. Pombal et al. noted in Tower v2: Unbabel-IST 2024 Submission for the General MT Shared Task, executing these multiple forward passes through external neural evaluation models introduces a prohibitive latency penalty.
3. The Quadratic Cost of Consensus
The latency bottleneck becomes even more pronounced when pipelines incorporate consensus-based selection techniques like Minimum Bayes Risk (MBR) decoding. MBR finds the candidate that minimizes expected risk by performing pairwise quality comparisons across the entire pool. If a system generates $N$ candidates, MBR requires $O(N^2)$ evaluations. As highlighted by Keito Kudo et al. in SKIM at WMT 2023 General Translation Task, combining Quality Estimation (QE) models with MBR decoding achieves remarkable accuracy but creates an exceptionally slow pipeline that is impractical for time-sensitive production environments.
A Pervasive Bottleneck Across Tasks
This tension between accuracy and inference speed is a recurring theme across diverse natural language processing domains:
- Machine Translation: In production settings like live chat translation, systems must operate under strict real-time constraints. Ana C. Farinha et al. showed in Findings of the WMT 2022 Shared Task on Chat Translation that while massive model ensembles and multi-stage rerankers achieve peak benchmark scores, they consistently violate the latency budgets required for live human interaction. Oleksii Kuchaiev et al. documented a similar barrier in NVIDIA NeMo's Neural Machine Translation Systems for English-German and English-Russian News and Biomedical Tasks at WMT21, noting that noisy-channel reranking creates a major deployment bottleneck.
- Dialogue State Tracking: In BREAK: Breaking the Dialogue State Tracking Barrier with Beam Search and Re-ranking, Seungpil Won et al. demonstrated that combining large-scale beam search with secondary rerankers multiplies inference time by several iterations compared to standard greedy search, severely limiting real-time dialogue feasibility.
- Structured Generation (Text-to-SQL): When generating code under ambiguity, advanced decoding strategies must explore structural variations. Adithya Bhaskar et al. showed in Benchmarking and Improving Text-to-SQL Generation under Ambiguity that executing structural plan perturbations requires significantly more sequential decoding steps, leading to high computational overhead.
- Decomposable Sequence Tasks: Even within single, end-to-end differentiable networks, executing sequential search operations at multiple intermediate stages (such as performing intermediate beam search for speech translation) compounds inference latency, as detailed by Siddharth Dalmia et al. in Searchable Hidden Intermediates for End-to-End Models of Decomposable Sequence Tasks.
Baking Reranking into a Single Pass
To circumvent the multi-pass tax, recent research has shifted toward a new paradigm: distilling the intelligence of multi-stage pipelines directly into the base model's weights. The goal is to achieve the quality gains of reranking or MBR decoding within a single, fast inference pass.
Multi-Stage (Slow): [Base Model] ──> [N Candidates] ──> [External Reranker] ──> [Output]
Single-Pass (Fast): [Aligned/Calibrated Model] ───────────────────────────────> [Output]
Direct Preference Optimization (DPO)
Instead of running expensive MBR selection at runtime, the model can be trained to inherently prefer high-consensus outputs. In Direct Preference Optimization for Neural Machine Translation with Minimum Bayes Risk Decoding, Guangyu Yang et al. utilized DPO to "bake" MBR benefits directly into the model. By training the network on pairs of high- and low-risk translations generated offline, the authors achieved MBR-level translation quality using standard, single-pass decoding.
Confidence Calibration
Another approach addresses the root cause of why reranking is necessary: poorly calibrated token probabilities. If a model's internal confidence does not correlate with actual output quality, it will select suboptimal tokens during standard decoding. Di Wu et al. tackled this in Calibrating Translation Decoding with Quality Estimation on LLMs by optimizing the Pearson correlation between a model's internal token confidence and external quality metrics. The resulting calibrated model can self-evaluate effectively, allowing efficient beam search to yield the same accuracy as an expensive external reranking pipeline.
Dynamic Hesitation
Rather than expanding the search space uniformly for every token or sentence, models can allocate computational budgets dynamically based on difficulty. Romain Storaï introduced a training-free, plug-and-play method called HARP in HARP: Hesitation-Aware Reframing in Transformer Inference Pass. HARP monitors token-level uncertainty during the inference pass. When the model "hesitates" on a difficult token, the system dynamically reframes the input using embedding dropout, allowing the model to rethink its choice on the fly. This targeted intervention provides performance gains while remaining significantly faster than traditional multi-candidate search methods.
Key Foundational Work
NVIDIA NeMo's Neural Machine Translation Systems for English-German and English-Russian News and Biomedical Tasks at WMT21 (2021) — contributed 2 concept elements to this direction.
Energy-Based Reranking: Improving Neural Machine Translation Using Energy-Based Models (2020) — contributed 1 concept elements to this direction.
Searchable Hidden Intermediates for End-to-End Models of Decomposable Sequence Tasks (2021) — contributed 1 concept elements to this direction.