Training Autoregressive Models on Self-Generated Sequences

83 concepts71 papers1 proposal

Autoregressive models excel at generating sequential data—whether it is text, video frames, or audio patches—by predicting the next token given a history of previous tokens. However, a fundamental flaw lies in how these models are traditionally trained versus how they are deployed. This discrepancy creates a compounding error loop known as exposure bias.

To eliminate this bottleneck, a prominent research direction focuses on shifting models away from perfect training data and instead forcing them to train on their own self-generated sequences.

The Teacher-Forcing Trap

During standard training, sequence models use a technique called teacher forcing. When predicting the token at step $t$, the model is given the perfect, ground-truth history of tokens from steps $1$ to $t-1$.

Training (Teacher Forcing):
Ground Truth:  The -> cat -> sat -> on -> the -> mat
Model Inputs:  The -> cat -> sat -> on -> the

Inference (Autonomous):
Model Outputs: The -> dog -> barked -> [Out of Distribution]

At inference time, the teacher disappears. The model must generate tokens autonomously, feeding its own previous predictions back into itself as context for the next step.

This creates a severe distribution mismatch. If the model makes a minor mistake early in generation, the input context shifts into a state it has never encountered during training. Because it has only ever seen perfect prefixes, it does not know how to handle an imperfect history. The initial error propagates and compounds exponentially, causing text to dissolve into repetitive loops or hallucinations (Wang), and causing autoregressive videos to degrade into visual drift or static (Huang et al., Liu et al.).

The Core Mechanism: Self-Rollout and Error Exposure

The solution is to align the training environment with the inference environment. Instead of feeding the model ground-truth tokens, the training loop incorporates self-rollouts, where the model's own top predictions or sampled outputs are fed back into the network as subsequent inputs.

Early foundations in structured prediction, such as the SEARN meta-algorithm introduced by Langford et al., demonstrated that training a classifier on its own predicted trajectories allows it to handle the exact states it will face during deployment. In modern deep learning, this manifests as exposing the model to its own iterative predictions during the optimization phase (Duan et al.).

By generating sequences based on its own history during training, the model's optimization objective directly accounts for the uncertainty and variance it will experience when running autonomously.

Why It Works: Learning to Recover

The core intuition behind training on self-generated sequences is not that it prevents the model from making mistakes, but that it teaches the model how to recover from them.

When a model is trained exclusively via teacher forcing, it learns a policy that is highly accurate on a narrow ridge of perfect data. However, it lacks any "steering correction" vectors outside of that ridge. By intentionally exposing the model to its own errors, the training process maps out the surrounding state space.

Teacher Forcing:
[Perfect Ridge] -------------------------> Success
                 (Mistake -> Fall off cliff)

Self-Generated Training:
[Perfect Ridge] -------------------------> Success
    \                                     /
     \--> [Imperfect State] ------------>/ (Recovery Path)

For example, Norouzi et al. developed Optimal Completion Distillation, which explicitly uses dynamic programming to find optimal paths to a target from any arbitrary mistake the model makes during a self-rollout. Similarly, Wang demonstrated that removing teacher forcing forces the model to develop internal correction strategies, making it robust against domain shifts where errors are highly likely.

Algorithmic Frameworks

Researchers have implemented self-generated sequence training across several distinct paradigms:

Reinforcement Learning and Sequence-Level Rewards

Reframing generation as a sequential decision-making problem allows the use of reinforcement learning (RL) to optimize directly for sequence-level metrics like BLEU or CIDEr (Shi, Ren et al.). A landmark implementation is Self-Critical Sequence Training (SCST) by Rennie et al., which uses the model's own greedy inference output as a baseline to evaluate its sampled rollouts, regularizing the model against its own typical mistakes.

On-Policy Knowledge Distillation

When compressing large language models, standard distillation suffers from exposure bias because the student model is evaluated on the teacher's data distribution. To fix this, Agarwal introduced Generalized Knowledge Distillation (GKD), which runs the student model on-policy. The student generates its own sequences, and the teacher provides feedback on those specific student-generated paths. This strategy is mirrored in frameworks like MiniLLM (Gu et al.) and PromptKD (Jang et al.), ensuring the student learns to stabilize its own distribution.

Differentiable Sampling and Soft Alignment

Because sampling discrete tokens is non-differentiable, some approaches use Gumbel-Softmax distributions to pass model-generated tokens downstream while maintaining gradient flow (Xu et al., Shao et al.). Others bypass token-level matching entirely, using soft semantic similarity to align self-generated hypotheses with reference targets (Stefánik et al.).

Application Beyond Text

While initially designed for natural language generation, mitigating exposure bias via self-rollouts has become essential across diverse modalities:

  • Autoregressive Video Diffusion: Models generating long-duration videos suffer from severe quality collapse over time. By utilizing self-generated frame rollouts during training, models like those designed by Huang et al. and Liu et al. learn to tolerate and correct minor visual artifacts, preventing compounding drift.
  • Speech Generation: In long-form text-to-speech synthesis, Zhuang et al. utilize a patch-based draft-and-refine architecture to maintain acoustic stability over extended horizons.
  • Structural Biology: In multi-stage pipelines like antibody design, the authors of the ASSD framework demonstrated that training downstream structural prediction models on the imperfect outputs of upstream sequence generators—rather than ground-truth structures—drastically improves the robustness of the entire system at inference time.

Key Foundational Work

Minimum Risk Training for Neural Machine Translation (2016) — contributed 1 concept elements to this direction.
Search-based structured prediction (2009) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_exposure_bias_mitigation_e28074ae,
  title = {exposure bias mitigation},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/exposure-bias-mitigation/rdr_e28074ae4e454f409da174165e307106},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_exposure_bias_mitigation_e28074ae_v1,
  title = {exposure bias mitigation},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/exposure-bias-mitigation/rdr_e28074ae4e454f409da174165e307106/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake