Autoregressive sequence generation models—whether used for machine translation, speech recognition, or text generation—share a fundamental design pattern: they generate text token-by-token from left to right (L2R). Mathematically, they decompose the joint probability of a sequence into a chain of conditional probabilities:
$$P(Y) = \prod_{i=1}^{N} P(y_i \mid y_{<i})$$
While computationally convenient, this unidirectional approach suffers from a severe structural flaw: directional bias. Because each token depends entirely on past choices, early mistakes cascade down the line. By the time a model reaches the end of a long sequence, its predictive quality often degrades significantly.
To overcome this limitation, a prominent line of research introduces Right-to-Left (R2L) reranking (or bidirectional sequence scoring). Instead of trusting a single left-to-right pass, these systems validate generated text using an independent model trained to evaluate sequences in the exact reverse direction.
The Problem: Asymmetric Error Accumulation
In standard sequence generation, errors are not distributed evenly. As Jun Suzuki notes in NTT Neural Machine Translation Systems at WAT 2019, standard decoders accumulate errors as the sentence progresses, lacking any mechanism to verify the consistency of the generated text from the opposite direction.
This directional bias manifests in two main ways:
- Cascading Early Errors: A mistake made in the first few tokens alters the conditioning context for all subsequent tokens, derailing the entire sequence (Yunzhe Yuan et al. in Bidirectional Transition-Based Dependency Parsing).
- Weak Suffixes: Because the end of the sentence is furthest from the initial conditioning prompt, the model's generation quality degrades toward the final tokens (Xu Tan et al. in Almost Unsupervised Text to Speech and Automatic Speech Recognition; Yuguang Wang et al. in Sogou Neural Machine Translation Systems for WMT17).
The Solution: Dual-Direction Validation
The core mechanism of R2L reranking is elegant: it decouples the generation of candidate sequences from their final evaluation.
[Input Prompt] ---> (L2R Model Beam Search) ---> [Top-N Candidates]
|
v
[Final Output] <--- (Combine Scores) <--- (R2L Model Backward Scoring)
First, a standard L2R model uses beam search to generate a list of the top-$N$ most likely candidate sequences. Next, a separate model trained to predict sequences in reverse order (right-to-left) scores the likelihood of those exact same candidates. Finally, the system combines the forward and backward scores to select the best output.
This approach has proven highly effective across diverse sequence transduction tasks, including machine translation (Rico Sennrich et al. in Edinburgh Neural Machine Translation Systems for WMT 16; Matthias Huck et al. in LMU Munich's Neural Machine Translation Systems at WMT 2018), name transliteration (Dayne Freitag in Name Transliteration with Bidirectional Perceptron Edit Models), and dialogue response generation (Milica Gasic et al. in Semantically Conditioned LSTM-based Natural Language Generation for Spoken Dialogue Systems).
Why It Works: The Intuition of Complementary Errors
To understand why this simple rescoring step yields such consistent performance gains, we must look at the error profiles of the two models.
An L2R model and an R2L model do not make the same mistakes. For an L2R model, the beginning of a sentence (the prefix) is easy because it is close to the source context, while the end (the suffix) is error-prone. For an R2L model, the situation is exactly reversed: the suffix is its starting point (highly accurate), while the prefix is generated last (error-prone).
As the authors of English-to-Japanese Diverse Translation by Combining Forward and Backward Outputs observe, forward models excel at generating sentence prefixes, whereas backward models are more effective at capturing sentence suffixes. By combining their perspectives, the high-confidence regions of one model compensate for the low-confidence regions of the other (Yunzhe Yuan et al. in Bidirectional Transition-Based Dependency Parsing).
Furthermore, Minwei Feng et al. demonstrate in The RWTH Aachen System for NTCIR-10 PatentMT that decoding sequences in reverse generates an entirely distinct set of structural errors. When you ensemble or combine these models, their consensus acts as a powerful filter that eliminates idiosyncratic, direction-specific mistakes.
Enforcing Global Consistency and Future Context
Standard autoregressive models are inherently short-sighted; they cannot look into the future. When an L2R model selects a word at step $t$, it cannot know if that choice will force it into a grammatical dead end at step $t+10$.
R2L reranking solves this by serving as a global consistency check (Dayne Freitag in Name Transliteration with Bidirectional Perceptron Edit Models). When the backward model evaluates a fully formed candidate sequence, it naturally incorporates "future" context relative to the forward pass.
As Wayne Xiong et al. explain in The microsoft 2016 conversational speech recognition system, reverse-order predictions significantly reduce error rates by integrating succeeding context into the final rescoring process. This backward-looking validation ensures that early lexical choices are compatible with the entire rest of the sequence (Deyi Xiong et al. in Enhancing Language Models in Statistical Machine Translation with Backward N-grams and Mutual Information Triggers), effectively catching local ambiguities that a unidirectional decoder would overlook (Aarne Talman et al. in The University of Helsinki Submissions to the WMT19 News Translation Task).
Beyond Post-Hoc Reranking
While reranking $n$-best lists is the most common implementation, the intuition of bidirectional validation has expanded into more tightly integrated architectures:
- Non-Autoregressive Refinement: In frameworks like Mask CTC, Shinji Watanabe et al. use bidirectional context to iteratively fill in low-confidence blanks, leveraging both preceding and succeeding tokens simultaneously to bypass the speed bottlenecks of pure autoregression.
- Iterative Correction: In information extraction tasks, Lu Dai et al. introduce Bi-Directional Iterative Prompt-Tuning for Event Argument Extraction, which processes sequences back-and-forth iteratively. By aggregating confidence scores from both directions dynamically, the system can identify and correct errors that would otherwise become permanent during a single-pass sequential inference.
Key Foundational Work
Name Transliteration with Bidirectional Perceptron Edit Models (2009) — contributed 2 concept elements to this direction.
Bidirectional Phrase-based Statistical Machine Translation (2009) — contributed 1 concept elements to this direction.
Syntax-Driven Machine Translation as a Model of ESL Revision (2010) — contributed 1 concept elements to this direction.