Standard autoregressive language models operate on a simple principle: given a prefix, predict the single next token. While this local objective has powered the modern era of large language models, it suffers from a fundamental flaw known as autoregressive myopia. Because the training objective only measures success one step at a time, models are heavily incentivized to exploit local statistical shortcuts—such as immediate syntax patterns and $n$-gram frequencies—rather than developing long-term plans or global structural representations.
To build intuition for why this happens, consider a task that requires lookahead, such as path-finding or complex code generation. Under standard teacher-forcing, a model is always fed the ground-truth prefix during training. As argued by Bachmann et al. in The Pitfalls of Next-Token Prediction, this creates an optimization environment where the model can "cheat." The model does not receive a sufficient gradient signal to learn the complex, early decision-making required for long-term planning because subsequent tokens are easily inferred from the ground-truth prefix anyway. Consequently, the model prioritizes short-term statistical patterns over global planning, leading to severe failures in tasks requiring combinational logic or lookahead, even within the training distribution.
This lookahead learning deficit manifests in several ways:
- Inability to navigate constraints: Without a mechanism to anticipate future consequences, models easily fall into paths where violating a long-term structural or thematic constraint becomes inevitable (Claveau et al., PPL-MCTS: Constrained Textual Generation Through Discriminator-Guided MCTS Decoding).
- Data inefficiency: Models require vast amounts of data to generalize global structures because they are forced to learn them implicitly through sparse local observations (Nagarajan et al., Roll the dice & look before you leap: Going beyond the creative limits of next-token prediction).
- Inability to answer global queries: Standard autoregressive representations lack native mechanisms to reason over entire future trajectories, such as predicting event ordering or hitting times (Boyd et al., Predictive Querying for Autoregressive Neural Sequence Models).
To resolve this, a growing body of research focuses on altering the training objective or decoding mechanisms to explicitly incentivize models to look ahead.
Multi-Token Prediction: Forcing Future-Aware Representations
The most direct way to break next-token myopia is to force the model to predict multiple future tokens simultaneously.
Standard Objective: [Token 1] [Token 2] [Token 3] ---> Predict [Token 4]
Multi-Token (n=3): [Token 1] [Token 2] [Token 3] ---> Predict [Token 4], [Token 5], [Token 6]
Why does this work? When a model's current hidden state must independently produce predictions for the next $n$ tokens, that hidden state can no longer rely solely on local bigram or trigram shortcuts. It is forced to encode macro-level features—such as structural dependencies, semantic goals, and algorithmic logic—that remain valid across a longer horizon.
Early foundations for this approach were laid by Qi et al. in ProphetNet: Predicting Future N-gram for Sequence-to-Sequence Pre-training, which introduced an "$n$-stream self-attention" mechanism to predict future $n$-grams simultaneously, preventing the model from overfitting to local word patterns. This architecture was later scaled across diverse domains and languages in ProphetNet-X (Qi et al.). Similarly, Goodman et al. developed TeaForN: Teacher-Forcing with N-grams, stacking decoders so that gradients could flow across multiple future timesteps during training, explicitly optimizing the internal representations for multi-step planning.
More recently, Gloeckle et al. demonstrated in Better & Faster Large Language Models via Multi-token Prediction that this objective significantly improves sample efficiency and performance on complex reasoning tasks like coding. While powerful, implementing multi-token prediction can be challenging for smaller architectures; to address this, Aynetdinov et al. proposed a curriculum-based training approach in Pre-Training Curriculum for Multi-Token Prediction in Language Models to smoothly transition models from local to global objectives.
Variations on the Multi-Token Theme
Rather than simply predicting the immediate next $n$ adjacent tokens, researchers have introduced variations to capture different types of structural dependencies:
- Leap Prediction: Liu et al. developed L-MTP: Leap Multi-Token Prediction Beyond Adjacent Context for Large Language Models, which trains models to predict non-sequential future tokens (e.g., positions $+1, +3, +5, +7$). This "leaping" mechanism forces the model to capture long-range structural skeletons rather than getting bogged down in immediate adjacent transitions.
- Token-Bag Exploitation: Instead of forcing the model to predict exact tokens at exact future positions, Liu et al. introduced Next-ToBE: Probabilistic Next Token-Bag Exploitation for Activating Anticipatory Capacity in LLMs. This method rewards the model for predicting an unordered, weighted "bag" of future tokens, activating its latent anticipatory capacity without the rigid penalty of exact positional alignment.
Latent Space and Summary Lookahead: Abstracting the Future
Predicting exact future tokens can be highly inefficient and noisy because there are many valid ways to phrase a future sentence. A parallel line of work solves this by forcing models to predict abstract representations or summaries of the future instead of raw tokens.
By shifting the prediction target to the latent space, the model is forced to build a compact "world model" or "belief state" that discards token-level noise but retains the overarching semantic trajectory.
Token-level: [Prefix] ---> Predict: "will", "go", "to", "the", "store"
Latent-level: [Prefix] ---> Predict: [Embedding of future shopping action]
For instance, Dai et al. introduced Context-level Language Modeling by Learning Predictive Context Embeddings (ContextLM), which trains models to predict "next-context" embeddings—latent representations of future multi-token chunks. This shift allows the model to capture broad semantic structures and long-range dependencies far more efficiently than standard autoregressive scaling.
In Next-Latent Prediction Transformers Learn Compact World Models, Tomar introduced an auxiliary objective (NextLat) where transformers learn to predict their own future internal representations. This self-prediction forces the network to compress its history into a robust internal world model. Similarly, Hu et al. designed The Belief State Transformer, which trains a model to simultaneously predict the next token of a prefix and the previous token of a suffix. This bidirectional constraint forces the transformer to learn a compact "belief state" that explicitly bridges the past to a distant future goal.
When a text sequence follows strict macro-level logic—such as step-by-step recipes or scientific processes—standard token prediction fails to capture how individual actions accumulate. To address this, Nandy et al. developed CLMSM: A Multi-Task Learning Framework for Pre-training on Procedural Text, utilizing a Masked Step Modeling objective to ensure the model tracks how entities change across an entire procedure. Taking lookahead to its logical conclusion, Mahajan et al. introduced Beyond Multi-Token Prediction: Pretraining LLMs with Future Summaries (FSP), a pretraining objective where models are explicitly tasked with generating a compressed summary of the long-term future context, significantly boosting long-horizon reasoning and coding capabilities.
Lookahead via Search and Distillation
When altering the foundational pretraining objective is not feasible, researchers have successfully mitigated myopia during decoding or through targeted knowledge distillation.
During inference, standard autoregressive generation can be augmented with lookahead search. As demonstrated in PPL-MCTS (Claveau et al.), treating text generation as a tree exploration process guided by a discriminator allows a frozen model to evaluate the future consequences of its current token choices, effectively preventing constraint violations before they occur.
Finally, the lookahead deficit presents a unique challenge for knowledge distillation. Standard distillation objectives focus on token-by-token probability matching, meaning a student model only learns the teacher's immediate next-token preferences, missing the underlying long-term planning. In Adversarial Moment-Matching Distillation of Large Language Models, Jia bypassed this limitation by framing distillation as an adversarial imitation learning problem. By training student models to match the "action-value moments" of the teacher, the student explicitly learns which early decisions contribute most to the long-term success and quality of the final generated response.
Key Foundational Work
ProphetNet: Predicting Future N-gram for Sequence-to-Sequence Pre-training (2020) — contributed 1 concept elements to this direction.
ProphetNet-X: Large-Scale Pre-training Models for English, Chinese, Multi-lingual, Dialog, and Code Generation (2021) — contributed 1 concept elements to this direction.
TeaForN: Teacher-Forcing with N-grams (2020) — contributed 1 concept elements to this direction.