Autoregressive models generate sequences step-by-step, conditioning each new prediction on the history of what came before. While highly effective, these models suffer from a fundamental vulnerability during deployment: a single early mistake can cause the entire generation to derail. Scheduled sampling is a training strategy designed to fix this fragility by gradually forcing models to learn from and correct their own errors.
The Training-Inference Gap: Exposure Bias
To understand why scheduled sampling works, we must first look at how sequence models are traditionally trained. The standard approach is a technique called teacher forcing. During training, even if the model makes an incorrect prediction at step $t$, it is given the true, ground-truth token as context for step $t+1$. It is as if a teacher is constantly correcting the student after every word, ensuring they never lose their place. This allows training to be highly parallelized and efficient.
However, this creates a severe mismatch when the model is deployed for real-world inference. Without a teacher present, the model must feed its own previous predictions back into itself as context for the next step.
Training (Teacher Forcing):
Ground Truth: The -> cat -> sat -> on -> the -> mat
Model Input: The -> cat -> sat -> on -> the -> mat
Inference (Autoregressive):
Model Input: The -> dog -> barked -> at -> the -> mailman
If the model makes a minor mistake early on—such as predicting "dog" instead of "cat"—the input distribution shifts away from the clean, perfect data it saw during training. Because the model was never exposed to its own imperfect outputs during training, it does not know how to recover. Errors compound rapidly, leading to a total breakdown in generation quality. This phenomenon is known as exposure bias.
The Core Mechanism: Mixing Truth and Predictions
Scheduled sampling bridges this gap by altering the training distribution to mimic real-world inference conditions. Instead of always feeding the ground-truth token, the training procedure flips a biased coin at each step. With probability $p$, the model receives the ground-truth token; with probability $1-p$, it receives its own prediction from the previous step.
Crucially, the probability $p$ is governed by a schedule or curriculum that changes over time:
- Early Training ($p \approx 1.0$): The model relies almost entirely on teacher forcing. When the model's predictions are still random noise, feeding them back into the network would make learning impossible (the "cold-start" problem).
- Late Training ($p \to 0.0$): As the model matures and its predictions become more accurate, the schedule reduces $p$. The model is increasingly forced to navigate its own self-generated sequences, learning how to handle and recover from minor deviations.
By gradually shifting the input distribution from idealized training conditions to practical deployment conditions, the model develops internal error-correction strategies.
Advanced Scheduling Strategies
Researchers have introduced several sophisticated variations to optimize this curriculum, moving beyond simple global linear decays:
- Functional Annealing: The transition can be shaped using non-linear functions to optimize learning stability. For example, Heng-Da Xu et al. in Generating Informative Dialogue Responses with Keywords-Guided Networks utilize a cosine annealing schedule to smoothly transition from ground-truth keywords to predicted ones, preventing the model from becoming overly dependent on perfect labels.
- Position-Dependent Schedules: Errors naturally accumulate more heavily as a sequence grows longer. To address this, Fandong Meng et al. in Scheduled Sampling Based on Decoding Steps for Neural Machine Translation introduced a schedule based on the sequence index. By increasing the reliance on self-generated predictions as the decoding step progresses, the training process precisely simulates the temporal error accumulation encountered during inference.
- Oracle and Mixed Sampling: Instead of forcing a binary choice between the exact ground truth or a potentially wild model prediction, some approaches sample "oracle" tokens—predictions that are close or semantically similar to the ground truth. Wen Zhang 0009 et al. in Bridging the Gap between Training and Inference for Neural Machine Translation (Extended Abstract) use this strategy to smooth the transition and handle distribution shifts gracefully. Similarly, Alexandros Haliassos et al. in Pay Attention to CTC: Fast and Robust Pseudo-Labelling for Unified Speech Recognition alternate between teacher-forced inputs and standard autoregressive generation to maintain robustness while preserving parallel training efficiency.
Application Across Modalities
While originally popularized in natural language processing, scheduled sampling has become a foundational technique across any domain involving sequential or multi-stage prediction:
- Computer Vision and Motion: In structured vision tasks like human pose estimation, Georgia Gkioxari et al. in Chained Predictions Using Convolutional Neural Networks replace ground-truth joint locations with previous predictions to prevent error propagation across the prediction chain. For continuous physical sequences, Dario Pavllo et al. in Modeling Human Motion with Quaternion-based Neural Networks and Félix G. Harvey et al. in Recurrent transition networks for character locomotion apply scheduled sampling to stabilize long-term human motion and character animation rollouts.
- Speech and Audio Processing: In speech recognition, Chung-Cheng Chiu et al. in State-of-the-art Speech Recognition With Sequence-to-Sequence Models use it to make models robust to errors in long transcriptions. It also stabilizes multi-speaker environments (Xuankai Chang et al. in End-to-End Monaural Multi-speaker ASR System without Pretraining) and multi-stage audio generation cascades (Ye-Xin Lu et al. in MultiStage Speech Bandwidth Extension with Flexible Sampling Rate Control).
- Spatiotemporal and Physical Simulations: For long-horizon forecasting, such as traffic prediction, Rose Yu et al. in Diffusion Convolutional Recurrent Neural Network: Data-Driven Traffic Forecasting use a mix of ground-truth and predicted inputs to mitigate drift. Similarly, in scientific computing, Makoto Takamoto et al. in Learning Neural PDE Solvers with Parameter-Guided Channel Attention employ a scheduled transition to autoregressive training to ensure neural solvers remain stable during long physical rollouts.
By forcing models to navigate their own imperfect outputs during training, scheduled sampling breaks the brittle reliance on perfect inputs, transforming fragile sequence predictors into robust, error-resilient systems.
Key Foundational Work
Supertagging With LSTMs (2016) — contributed 2 concept elements to this direction.
Chained Predictions Using Convolutional Neural Networks (2016) — contributed 1 concept elements to this direction.
Lip Reading Sentences in the Wild (2016) — contributed 1 concept elements to this direction.