In many machine learning tasks, we don't just predict a single value; we predict a structure. Whether it is a sequence of words in a translation, a set of tags for a document, or the coordinates of joints in a human body, these outputs are rarely independent. To capture these dependencies, researchers often use sequential prediction—an approach where the model predicts one element at a time, using its own previous outputs as context for the next decision.
While intuitive, this "predictions-as-features" approach harbors a fundamental flaw: the snowball effect of error propagation. If the model makes a single mistake early in the sequence, that error becomes part of the "ground truth" context for every subsequent step, leading to a catastrophic collapse in accuracy.
The Exposure Bias Problem
The root of error propagation lies in a discrepancy between how models are trained and how they are used. During training, models are typically "teacher-forced"—they are given the actual ground-truth labels of previous steps to predict the next one. However, during inference, the teacher is gone. The model must rely on its own noisy, potentially incorrect predictions.
Robin Senge and Juan José del Coz demonstrate in Rectifying Classifier Chains for Multi-Label Classification that the probability of a downstream classifier receiving noise-free input decreases exponentially as a chain grows longer. This "attribute noise" means that later classifiers in a sequence are often trying to make sense of a context that is fundamentally corrupted. This is visible across domains:
- Computer Vision: In human pose estimation, Georgia Gkioxari and colleagues found that if an initial prediction for a shoulder is wrong, the subsequent prediction for the elbow—which is conditioned on the shoulder—will almost certainly be misplaced (Chained Predictions Using Convolutional Neural Networks).
- Natural Language Processing: In entity linking, Xiyuan Yang notes that resolving an ambiguous mention incorrectly "poisons" the context for all remaining entities in the document (Learning Dynamic Context Augmentation for Global Entity Linking).
Strategic Reordering: The Easy-First Principle
If the order of prediction determines the risk of error, can we simply choose a better order? Traditional models process data in a fixed direction (e.g., left-to-right). However, some parts of a problem are inherently easier than others.
Andre Martins proposed "Easy-First" taggers that learn to label the most certain tokens first, regardless of their position in the sentence (Learning What's Easy: Fully Differentiable Neural Easy-First Taggers). By building a foundation of high-confidence labels, the model provides a cleaner context for the "hard" tokens later on. Similarly, Libin Shen and Aravind K. Joshi showed that dynamically determining the labeling order can significantly reduce error rates in part-of-speech tagging (Guided Learning for Bidirectional Sequence Classification).
More recently, Bohan Feng and Tenghsien Lai have explored reordering target sequences based on Mutual Information, arguing that predicting "information-rich" tokens too early—before the model has enough context—leads to hallucinations (Training LLMs Beyond Next Token Prediction - Filling the Mutual Information Gap).
Correction and Refinement Loops
Another way to break the error chain is to allow the model to change its mind. In standard autoregressive models, once a token is generated, it is immutable. Synchronous Refinement, introduced by Kehai Chen and others, allows a decoder to look back and update previous hidden states as it generates new ones, effectively "fixing" early mistakes in real-time (Synchronous Refinement for Neural Machine Translation).
In the realm of image segmentation, Shin'ichi Satoh and Ziling Huang utilize a Progressive Alignment Network to filter out errors between stages of a cascaded framework, ensuring that only high-quality features are propagated forward (Referring Image Segmentation via Joint Mask Contextual Embedding Learning and Progressive Alignment Network).
Beyond Linear Chains
When dependencies are too complex for a simple sequence, researchers move toward more robust structures:
- Joint Learning: Houfeng Wang and colleagues developed methods where "future" labels can influence "past" labels during training, capturing bidirectional dependencies that a one-way chain would miss (Multi-label Text Categorization with Joint Learning Predictions-as-Features Method).
- Multi-view Aggregation: Instead of relying on one "correct" order, Zhibin Gou suggests predicting elements in multiple different orders and using a voting mechanism to find the consensus, which prevents a single early error from ruining the entire output (MvP: Multi-view Prompting Improves Aspect Sentiment Tuple Prediction).
- Latent Independence: In some cases, the best way to handle dependency is to avoid modeling it explicitly. Teng Huang proposes learning a latent factor that makes different classification dimensions conditionally independent, thereby "escaping" the risk of error propagation entirely (Towards Escaping from Class Dependency Modeling for Multi-Dimensional Classification).
By acknowledging that sequential models are fragile, these diverse research directions share a common goal: building systems that are robust enough to survive their own inevitable mistakes.