Error Propagation in Contextual Sequence Generation

27 concepts19 papers1 proposal

Modern sequence models—whether for speech recognition, machine translation, or grammatical correction—increasingly rely on long-range context to improve accuracy. By "looking back" at previous utterances or sentences, these systems can resolve ambiguities in pronouns, verb tense, and topic coherence. However, this dependency introduces a critical vulnerability: contextual error propagation. When a model uses its own previous, potentially incorrect outputs as context for future predictions, a single mistake can "poison" the history, leading to a cascade of subsequent errors.

The Feedback Loop of Contextual Poisoning

The fundamental intuition behind this problem is the creation of a negative feedback loop. In tasks like conversational Automatic Speech Recognition (ASR), systems often use the predicted text of previous turns to inform the current sentence. As Xie et al. and Li et al. observe, if the system misidentifies a word in an early turn, that mistake provides flawed linguistic cues for the next turn.

This is not merely a localized issue. Abdullah et al. demonstrate that modern architectures are so sensitive to context that perturbing a single phoneme can trigger a cascading failure, causing the model to hallucinate an entirely different, incorrect narrative for the rest of the sentence. Flynn describes this as a "poisoning of the history," where the model becomes overly dependent on its own past mistakes, resulting in a sequence of transcriptions that are contextually consistent with the error but factually detached from the audio.

The Ground-Truth Gap

A recurring theme in the research is the discrepancy between how models are trained and how they perform in the real world. During training, models are often exposed to "gold-standard" human translations or transcriptions as context. However, during inference, they must rely on their own noisy hypotheses.

Lopes et al. and Pal et al. highlight a significant performance gap between using human-verified context and model-generated context. This "exposure bias" suggests that models are not inherently robust to the noise they generate. In document-level Neural Machine Translation (NMT), Hussein et al. and Herold et al. find that while long-context models excel at maintaining consistency (such as pronoun gender), the accumulation of early translation errors can eventually degrade the quality of the entire document.

Sensitivity Across Modalities

This sensitivity is not limited to text and speech. In sign language translation, Raajesh notes that using previous model outputs as context can pollute the reasoning for all following sentences in a continuous discourse. Even in error correction tasks, where the goal is to fix mistakes, the problem persists. Xu et al. show that Large Language Models (LLMs) used for ASR error correction can inadvertently increase error rates if they rely on flawed historical context, leading them to hallucinate corrections that match the incorrect history rather than the actual speech.

The effect is also visible in low-level acoustic modeling. Yu et al. argue that models incorporating long-span temporal context suffer from an "error-spreading" effect, where a single misidentified phonetic segment negatively impacts the acoustic likelihood of adjacent correct segments.

Strategies for Robustness

To combat these cascades, researchers have proposed several mechanisms to decouple the model from its own errors:

  • Source-Side Context: Wang et al. suggest that utilizing source-language history is more robust than relying on previously generated target-language sentences, as the source text is inherently "clean."
  • Cross-Modal Learning: Xie et al. propose extracting linguistic context directly from the audio of preceding utterances rather than their error-prone text transcripts, bypassing the textual feedback loop.
  • Error Exposure during Training: Duan et al. introduce strategies like "Error Exposure," which intentionally expose models to their own prediction errors during training to teach them how to recover from noisy contexts.
  • Multi-Step Refinement: Nakatani et al. advocate for a two-step approach where initial errors are reduced using local models (like N-grams) before applying long-range contextual features, ensuring the context itself is as accurate as possible.

While long-context models offer the potential for human-like coherence, the research by Liu et al. and Gaido et al. emphasizes that without robustness to historical noise, the benefits of additional information can be outweighed by the risks of contextual poisoning.

History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_contextual_error_propagation_8e5155a8,
  title = {contextual error propagation},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/contextual-error-propagation/rdr_8e5155a889c648c5afefbe836f536f4e},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_contextual_error_propagation_8e5155a8_v1,
  title = {contextual error propagation},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/contextual-error-propagation/rdr_8e5155a889c648c5afefbe836f536f4e/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake