Overcoming the Myopia of Next Token Prediction

26 concepts23 papers2 proposals

Standard autoregressive language models operate on a simple principle: given a prefix, predict the single next token. While this local objective has powered the modern era of large language models, it suffers from a fundamental flaw known as autoregressive myopia. Because the training objective only measures success one step at a time, models are heavily incentivized to exploit local statistical shortcuts—such as immediate syntax patterns and $n$-gram frequencies—rather than developing long-term plans or global structural representations.

To build intuition for why this happens, consider a task that requires lookahead, such as path-finding or complex code generation. Under standard teacher-forcing, a model is always fed the ground-truth prefix during training. As argued by Bachmann et al. in The Pitfalls of Next-Token Prediction, this creates an optimization environment where the model can "cheat." The model does not receive a sufficient gradient signal to learn the complex, early decision-making required for long-term planning because subsequent tokens are easily inferred from the ground-truth prefix anyway. Consequently, the model prioritizes short-term statistical patterns over global planning, leading to severe failures in tasks requiring combinational logic or lookahead, even within the training distribution.

This lookahead learning deficit manifests in several ways:

To resolve this, a growing body of research focuses on altering the training objective or decoding mechanisms to explicitly incentivize models to look ahead.


Multi-Token Prediction: Forcing Future-Aware Representations

The most direct way to break next-token myopia is to force the model to predict multiple future tokens simultaneously.

Standard Objective:  [Token 1] [Token 2] [Token 3] ---> Predict [Token 4]
Multi-Token (n=3):   [Token 1] [Token 2] [Token 3] ---> Predict [Token 4], [Token 5], [Token 6]

Why does this work? When a model's current hidden state must independently produce predictions for the next $n$ tokens, that hidden state can no longer rely solely on local bigram or trigram shortcuts. It is forced to encode macro-level features—such as structural dependencies, semantic goals, and algorithmic logic—that remain valid across a longer horizon.

Early foundations for this approach were laid by Qi et al. in ProphetNet: Predicting Future N-gram for Sequence-to-Sequence Pre-training, which introduced an "$n$-stream self-attention" mechanism to predict future $n$-grams simultaneously, preventing the model from overfitting to local word patterns. This architecture was later scaled across diverse domains and languages in ProphetNet-X (Qi et al.). Similarly, Goodman et al. developed TeaForN: Teacher-Forcing with N-grams, stacking decoders so that gradients could flow across multiple future timesteps during training, explicitly optimizing the internal representations for multi-step planning.

More recently, Gloeckle et al. demonstrated in Better & Faster Large Language Models via Multi-token Prediction that this objective significantly improves sample efficiency and performance on complex reasoning tasks like coding. While powerful, implementing multi-token prediction can be challenging for smaller architectures; to address this, Aynetdinov et al. proposed a curriculum-based training approach in Pre-Training Curriculum for Multi-Token Prediction in Language Models to smoothly transition models from local to global objectives.

Variations on the Multi-Token Theme

Rather than simply predicting the immediate next $n$ adjacent tokens, researchers have introduced variations to capture different types of structural dependencies:


Latent Space and Summary Lookahead: Abstracting the Future

Predicting exact future tokens can be highly inefficient and noisy because there are many valid ways to phrase a future sentence. A parallel line of work solves this by forcing models to predict abstract representations or summaries of the future instead of raw tokens.

By shifting the prediction target to the latent space, the model is forced to build a compact "world model" or "belief state" that discards token-level noise but retains the overarching semantic trajectory.

Token-level:   [Prefix] ---> Predict: "will", "go", "to", "the", "store"
Latent-level:  [Prefix] ---> Predict: [Embedding of future shopping action]

For instance, Dai et al. introduced Context-level Language Modeling by Learning Predictive Context Embeddings (ContextLM), which trains models to predict "next-context" embeddings—latent representations of future multi-token chunks. This shift allows the model to capture broad semantic structures and long-range dependencies far more efficiently than standard autoregressive scaling.

In Next-Latent Prediction Transformers Learn Compact World Models, Tomar introduced an auxiliary objective (NextLat) where transformers learn to predict their own future internal representations. This self-prediction forces the network to compress its history into a robust internal world model. Similarly, Hu et al. designed The Belief State Transformer, which trains a model to simultaneously predict the next token of a prefix and the previous token of a suffix. This bidirectional constraint forces the transformer to learn a compact "belief state" that explicitly bridges the past to a distant future goal.

When a text sequence follows strict macro-level logic—such as step-by-step recipes or scientific processes—standard token prediction fails to capture how individual actions accumulate. To address this, Nandy et al. developed CLMSM: A Multi-Task Learning Framework for Pre-training on Procedural Text, utilizing a Masked Step Modeling objective to ensure the model tracks how entities change across an entire procedure. Taking lookahead to its logical conclusion, Mahajan et al. introduced Beyond Multi-Token Prediction: Pretraining LLMs with Future Summaries (FSP), a pretraining objective where models are explicitly tasked with generating a compressed summary of the long-term future context, significantly boosting long-horizon reasoning and coding capabilities.


Lookahead via Search and Distillation

When altering the foundational pretraining objective is not feasible, researchers have successfully mitigated myopia during decoding or through targeted knowledge distillation.

During inference, standard autoregressive generation can be augmented with lookahead search. As demonstrated in PPL-MCTS (Claveau et al.), treating text generation as a tree exploration process guided by a discriminator allows a frozen model to evaluate the future consequences of its current token choices, effectively preventing constraint violations before they occur.

Finally, the lookahead deficit presents a unique challenge for knowledge distillation. Standard distillation objectives focus on token-by-token probability matching, meaning a student model only learns the teacher's immediate next-token preferences, missing the underlying long-term planning. In Adversarial Moment-Matching Distillation of Large Language Models, Jia bypassed this limitation by framing distillation as an adversarial imitation learning problem. By training student models to match the "action-value moments" of the teacher, the student explicitly learns which early decisions contribute most to the long-term success and quality of the final generated response.

Key Foundational Work

ProphetNet: Predicting Future N-gram for Sequence-to-Sequence Pre-training (2020) — contributed 1 concept elements to this direction.
ProphetNet-X: Large-Scale Pre-training Models for English, Chinese, Multi-lingual, Dialog, and Code Generation (2021) — contributed 1 concept elements to this direction.
TeaForN: Teacher-Forcing with N-grams (2020) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_local_objective_bias_706d36d1,
  title = {local objective bias},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/local-objective-bias/rdr_706d36d1b8be4a2daa950663ad340b03},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-19},
}

This exact version (version 1, frozen text):

@misc{lacuna_local_objective_bias_706d36d1_v1,
  title = {local objective bias},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/local-objective-bias/rdr_706d36d1b8be4a2daa950663ad340b03/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-19},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake