Scheduled Sampling for Autoregressive Sequence Models

105 concepts84 papers1 proposal

Autoregressive models generate sequences step-by-step, conditioning each new prediction on the history of what came before. While highly effective, these models suffer from a fundamental vulnerability during deployment: a single early mistake can cause the entire generation to derail. Scheduled sampling is a training strategy designed to fix this fragility by gradually forcing models to learn from and correct their own errors.

The Training-Inference Gap: Exposure Bias

To understand why scheduled sampling works, we must first look at how sequence models are traditionally trained. The standard approach is a technique called teacher forcing. During training, even if the model makes an incorrect prediction at step $t$, it is given the true, ground-truth token as context for step $t+1$. It is as if a teacher is constantly correcting the student after every word, ensuring they never lose their place. This allows training to be highly parallelized and efficient.

However, this creates a severe mismatch when the model is deployed for real-world inference. Without a teacher present, the model must feed its own previous predictions back into itself as context for the next step.

Training (Teacher Forcing):
Ground Truth:  The -> cat -> sat -> on -> the -> mat
Model Input:   The -> cat -> sat -> on -> the -> mat

Inference (Autoregressive):
Model Input:   The -> dog -> barked -> at -> the -> mailman

If the model makes a minor mistake early on—such as predicting "dog" instead of "cat"—the input distribution shifts away from the clean, perfect data it saw during training. Because the model was never exposed to its own imperfect outputs during training, it does not know how to recover. Errors compound rapidly, leading to a total breakdown in generation quality. This phenomenon is known as exposure bias.

The Core Mechanism: Mixing Truth and Predictions

Scheduled sampling bridges this gap by altering the training distribution to mimic real-world inference conditions. Instead of always feeding the ground-truth token, the training procedure flips a biased coin at each step. With probability $p$, the model receives the ground-truth token; with probability $1-p$, it receives its own prediction from the previous step.

Crucially, the probability $p$ is governed by a schedule or curriculum that changes over time:

  1. Early Training ($p \approx 1.0$): The model relies almost entirely on teacher forcing. When the model's predictions are still random noise, feeding them back into the network would make learning impossible (the "cold-start" problem).
  2. Late Training ($p \to 0.0$): As the model matures and its predictions become more accurate, the schedule reduces $p$. The model is increasingly forced to navigate its own self-generated sequences, learning how to handle and recover from minor deviations.

By gradually shifting the input distribution from idealized training conditions to practical deployment conditions, the model develops internal error-correction strategies.

Advanced Scheduling Strategies

Researchers have introduced several sophisticated variations to optimize this curriculum, moving beyond simple global linear decays:

Application Across Modalities

While originally popularized in natural language processing, scheduled sampling has become a foundational technique across any domain involving sequential or multi-stage prediction:

By forcing models to navigate their own imperfect outputs during training, scheduled sampling breaks the brittle reliance on perfect inputs, transforming fragile sequence predictors into robust, error-resilient systems.

Key Foundational Work

Supertagging With LSTMs (2016) — contributed 2 concept elements to this direction.
Chained Predictions Using Convolutional Neural Networks (2016) — contributed 1 concept elements to this direction.
Lip Reading Sentences in the Wild (2016) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_scheduled_sampling_5b62703f,
  title = {scheduled sampling},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/scheduled-sampling/rdr_5b62703f46a6424eb22edfd2eca67431},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_scheduled_sampling_5b62703f_v1,
  title = {scheduled sampling},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/scheduled-sampling/rdr_5b62703f46a6424eb22edfd2eca67431/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake