Breaking Repetitive Loops in Autoregressive Text Generation

You are reading version 1 of this direction, frozen 2026-10-04. The latest version is version 2.

Version 1 of 2511 papers596 conceptsfrozen 2026-10-04

When a Large Language Model (LLM) generates text, it does so one token at a time, predicting the next word based on everything it has written so far. Despite being trained on nearly the entire internet, these models frequently fall into a "degeneration" trap: they begin repeating the same phrase or sentence indefinitely. This phenomenon, often called a "repetition loop," is not just a minor glitch; it is a fundamental failure of the autoregressive framework that persists even as models scale to trillions of parameters.

The Geometry of a Repetition Trap

To understand why loops happen, we can view text generation as a walk through a directed graph of possible sequences. In an ideal world, the model would always find a path toward a natural conclusion. However, as Sean Welleck and colleagues demonstrate in Consistency of a Recurrent Language Model With Respect to Incomplete Decoding, many decoding algorithms are "inconsistent"—they can enter states where the probability of ever reaching an end-of-sequence token drops to zero.

The model essentially gets stuck in a "high-inflow" region of the probability landscape. Wai Lam and Zihao Fu provide a theoretical explanation in A Theoretical Analysis of the Repetition Problem in Text Generation, identifying specific words that act as "sinks." Once the model generates these words, the probability distribution shifts such that the most likely next tokens are the ones it just produced. This creates a self-reinforcing cycle where the model's own output becomes a "magnet" for its next prediction.

Why Scale Doesn't Fix It

It is tempting to assume that larger models or more data would solve this. However, Abigail See and Christopher D Manning found in Do Massively Pretrained Language Models Make Better Storytellers? that even massive models like GPT-2 remain prone to these traps. The issue lies in the training objective: Maximum Likelihood Estimation (MLE).

MLE rewards the model for assigning high probability to the "correct" next token, but it doesn't explicitly punish the model for assigning high probability to repetitive, nonsensical sequences. Furthermore, John Hewitt and Percy Liang argue in Truncation Sampling as Language Model Desmoothing that models often "hedge their bets" during training, placing tiny amounts of probability on many tokens to avoid infinite loss. During generation, these "noisy tails" can accumulate, eventually nudging the model off a coherent path and into a loop.

Researchers have developed two primary ways to break these loops: changing how we search the landscape (decoding) and changing the landscape itself (training).

The most famous intervention is Nucleus Sampling (top-p), introduced by Ari Holtzman in The Curious Case of Neural Text Degeneration. Instead of picking the single most likely word (greedy search), which often leads straight into a loop, Nucleus Sampling ignores the "tail" of the distribution and samples from the most likely tokens. This injects enough randomness to "jump" out of potential cycles.

More recently, Yixuan Su and Nigel Collier proposed Contrastive Search in A Contrastive Framework for Neural Text Generation. This method forces the model to pick a token that is both high-probability and visually distinct from the previous context, effectively penalizing the model for repeating itself in real-time.

2. Teaching What Not to Say

Instead of just fixing the search, we can train the model to avoid loops entirely. Sean Welleck and Jason E Weston introduced Unlikelihood Training in Neural Text Generation with Unlikelihood Training. This adds a secondary loss function that explicitly lowers the probability of tokens that have already appeared in the sequence. By teaching the model what not to say, it learns to build a probability landscape that is naturally resistant to loops.

Mechanistic Origins of Repetition

Recent work has moved from treating the model as a "black box" to finding the actual neural circuits responsible for these loops. In Repetition Neurons: How Do Language Models Produce Repetitions?, researchers identified specific "repetition neurons" that fire when a model begins to loop.

Building on this, Di Wang and colleagues used Sparse Autoencoders in Understanding the Repeat Curse in Large Language Models from a Feature Perspective to isolate "Repetition Features." They found that by surgically deactivating these features, they could stop a model from looping without damaging its ability to perform other tasks. This suggests that the "repeat curse" is not an inevitable part of language modeling, but a specific, isolatable failure mode of the transformer architecture.

Papers in this version

Showing 60 of 511 papers.

History & citationVersion 2 · 2026-10-04

Current · Permanent link

  1. Version 2 (latest) · 2026-10-04 · 353 papers
  2. Version 1 · 2026-10-04 · 511 papers
Cite this direction

Latest version (a living page):

@misc{lacuna_neural_text_degeneration_b299c780,
  title = {neural text degeneration},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/neural-text-degeneration/rdr_b299c78029ad4ec183280757b4ef9c1c},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_breaking_repetitive_loops_in_autoregress_b299c780_v1,
  title = {Breaking Repetitive Loops in Autoregressive Text Generation},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/breaking-repetitive-loops-in-autoregressive-text-generation/rdr_b299c78029ad4ec183280757b4ef9c1c/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}