TL;DR
Large Language Models (LLMs) have achieved remarkable success, largely due to the Next Token Prediction (NTP) paradigm. However, NTP also leads to critical weaknesses like poor long-term planning and error accumulation. This survey explores five emerging families of alternatives to NTP, aiming to guide research towards more capable and efficient text generation models.
What Problem Does This Paper Address?
The paper tackles the fundamental limitations of the dominant Next Token Prediction (NTP) approach in Large Language Models (LLMs). While NTP has enabled impressive fluency, it inherently struggles with tasks requiring long-term planning, structured reasoning, and global coherence. This myopic, token-by-token generation process leads to issues like:
- Error Accumulation: Small errors early in generation can compound, leading to nonsensical or factually incorrect outputs.
- Granularity Mismatch: LLMs operate on subword tokens, which don't align with human conceptual units like ideas or sentences, hindering higher-level reasoning.
- Computational Inefficiency: The quadratic complexity of attention mechanisms in Transformers, combined with token-level processing, makes generation computationally expensive.
- Misaligned Supervision: Teacher-forcing during training can lead models to learn "shortcuts" rather than true understanding.
The authors argue that these are not incidental failures but stem from the core NTP objective, motivating the exploration of alternative paradigms.
Methods
This survey categorizes and synthesizes research on alternatives to NTP into five main families:
- Multi-Token Prediction (MTP): Instead of predicting one token at a time, MTP models predict a block of k future tokens simultaneously. This is typically achieved using a shared "trunk" representation that feeds multiple parallel "heads," each predicting a future token. This aims to improve efficiency and short-term planning.
- Plan-then-Generate (PtG): This approach introduces a two-stage process. First, a high-level plan (either symbolic or latent) is generated from the input context. Second, this plan guides the subsequent token-level generation. This allows for global planning and revision.
- Latent Reasoning (LR): LR models shift the autoregressive process into a continuous latent space. Instead of predicting discrete tokens, they generate a sequence of latent vectors, which are then decoded into text. This addresses the granularity mismatch by operating on higher-level semantic units.
- Continuous Generation Approaches (CG): These methods abandon sequential, left-to-right generation entirely. They treat text generation as a global optimization problem, initializing the entire output and refining it iteratively through processes like diffusion, flow matching, or energy-based models. This enables global planning and bidirectional refinement.
- Non-Transformer Architectures (NTA): This category explores fundamentally different model architectures that bypass NTP by design. Examples include Recurrent Neural Networks (RNNs), State Space Models (SSMs), and Joint Embedding Predictive Architectures (JEPAs), which model sequences using different mechanisms like continuous states or shared semantic spaces.
Datasets
Not discussed in the paper.
Key Insights and Ideas
- NTP is the bottleneck, not the Transformer: The paper strongly suggests that the limitations of LLMs stem from the NTP objective itself, rather than inherent flaws in the Transformer architecture. Internal representations within Transformers often show sophisticated, multi-token abstractions, indicating models can reason beyond single tokens.
- A spectrum of alternatives: The survey reveals a rich and diverse ecosystem of approaches moving beyond NTP, ranging from simple extensions like MTP to radical departures like CG and NTA.
- Trade-offs are inherent: Each alternative family presents its own set of advantages (e.g., better planning, efficiency) and disadvantages (e.g., complexity, interpretability challenges, training difficulty).
- The future is likely hybrid: While each category offers unique benefits, the most promising future directions might involve combining elements from different families (e.g., compositional strategies for CG, hybrid planners for PtG).
Key Results
The paper is a survey and does not present new empirical results. Instead, it synthesizes findings from existing research. For example:
- MTP: Approaches like Joint Token Prediction (JTP) have shown improved performance on synthetic reasoning benchmarks and can enable speculative decoding, accelerating generation by up to 3x with minimal quality degradation.
- LR: SentenceVAE models have demonstrated significant improvements in inference speed (2-3x), perplexity (46-75%), and memory usage (86-91% reduction) compared to token-level baselines.
- CG: While generally underperforming strong autoregressive baselines in fluency and coherence, models like Mercury achieve impressive speeds (1100 tokens/sec) for specific tasks.
Limitations and Future Work
Limitations:
- Proprietary research: The survey acknowledges that its coverage might not include the most recent or unpublished advances, especially those from proprietary or closed-source settings.
- Blurred boundaries: The categories are high-level, and the boundaries between them can be blurred, with some models incorporating elements from multiple families.
- Focus on generation: The analysis primarily focuses on paradigms that challenge the generative process itself, with less emphasis on tokenization, output head optimizations, or intermediate architectural innovations.
- No empirical benchmarking: The survey relies on results reported by original authors, rather than presenting its own controlled empirical benchmarks.
Future Work:
- MTP: Predict farther ahead (increase k), use MTP as an auxiliary training method, and explore variable-length or adaptive horizons.
- PtG: Develop diverse planners that capture richer discourse structures, explore hybrid architectures of planning modules, and study adaptable time horizons.
- LR: Design architectures with a small "reasoning core" and lighter decoder for more internal thinking steps, improve encoders/decoders for text-to-latent space translation, and explore non-Transformer models for latent prediction.
- CG: Combine diffusion with higher-level planning (e.g., outlines), explore compositional strategies, and investigate adaptive step counts and block-wise refinement with cache reuse.
- NTA: Further explore SSMs and JEPAs, focusing on interpretability, generalization, and integration into practical generation systems.
Summary updated