Your LLM Knows the Future: Uncovering Its Multi-Token Prediction Potential

Mohammad Samragh, Arnav Kundu, David Harrison, Kumari Nishu, Devang Naik, Minsik Cho, Mehrdad Farajtabar

TL;DR

Autoregressive language models are notoriously slow because they generate text one token at a time. This paper introduces a novel framework that unlocks an LLM's latent ability to predict multiple future tokens simultaneously using masked inputs and a unique "Gated LoRA" mechanism. The result is a massive inference speedup—up to 5x in code and math generation—with absolutely zero loss in model quality.

What Problem Does This Paper Address?

Standard Large Language Models (LLMs) operate autoregressively: to generate a sentence, they predict the next token, append it to the prompt, and repeat. While highly effective, this sequential bottleneck makes inference computationally expensive and slow.

To accelerate generation, researchers have turned to Multi-Token Prediction (MTP) and speculative decoding. Speculative decoding typically relies on a separate, smaller "draft" model to guess future tokens, which adds architectural complexity. Direct MTP trains a single model to output multiple tokens at once, but fine-tuning an existing model for MTP often degrades its core Next-Token Prediction (NTP) accuracy—a phenomenon akin to catastrophic forgetting.

The authors of this paper ask a fundamental question: Can we train a single model to reliably predict multiple future tokens without sacrificing its original generation quality?

Methods

The authors propose a clean, modular architecture to turn a standard autoregressive LLM into an efficient multi-token predictor:

  • Masked-Input Formulation: Instead of adding heavy, complex prediction heads, the authors append $k$ special <mask> tokens to the end of a prompt. The model uses its full depth and capacity to fill in these masks with future tokens.
  • Gated LoRA: To protect the base model's original performance, the authors introduce a binary gating mechanism for low-rank adapters (LoRA). The LoRA path is activated ($I(t)=1$) only when processing <mask> tokens (MTP) and completely bypassed ($I(t)=0$) for regular text tokens (NTP). This ensures the original autoregressive capabilities remain completely untouched.
  • Lightweight Sampler Module: To ensure the predicted future tokens form a coherent sequence, they feed the model's hidden states into a lightweight, two-layer Multi-Layer Perceptron (MLP) sampler. This sampler conditions each predicted mask token on both the current context and the immediately preceding sampled token.
  • Latent Consistency Loss (LCM): This auxiliary self-distillation loss forces the hidden representations of early mask tokens to align closely with the actual future hidden states of the autoregressive model, improving prediction accuracy.
  • Quadratic Decoding: A speculative decoding strategy that interleaves mask tokens within speculative tokens in a tree-like structure. This ensures that even if a speculative token chain breaks early during verification, a steady stream of tokens is still available for verification at each step.
Figure 3: Components of our MTP model. Box-1 (top-left) shows the autoregressive model augmented with gated LoRA parameters. Box-2 (bottom left) illustrates the sampler head. Box-3 (right) presents the block diagram of the gated LoRA module.

Datasets

The framework was evaluated by fine-tuning a Tulu3-8B SFT model using the Tulu3 dataset, which contains nearly 1 million examples spanning math, code, conversation, and scientific benchmarks.

To measure speedups and verify that no quality degradation occurred, the authors tested the model across several prominent benchmarks:

  • Knowledge: MMLU, PopQA, TruthfulQA
  • Math & Coding: GSM8k, HumanEval
  • Chat: AlpacaEval, IFEval
  • Safety: XSTest, HarmBench, WildGuardTest, and others.

Key Insights and Ideas

  • The Future is Already There: Intriguingly, pretrained LLMs already implicitly know the future. When given placeholder tokens, a vanilla model ranks the correct future tokens within its top-200 logits. Fine-tuning with mask tokens simply coaxes this latent knowledge into the top-10.
Figure 1: Autoregressive models implicitly anticipate future tokens. (Left): A pretrained model queried with a prompt plus redundant (-) tokens ranks the correct next token within the top-200. (Middle): Finetuning with <mask> tokens improves structure, pushing correct predictions into the top-10. (Right): Training a sampler head further refines future token prediction.
  • Standard LoRA Degrades Capabilities: Fine-tuning an SFT model with standard LoRA causes a sharp drop in downstream task accuracy (e.g., ARC-Challenge). Gated LoRA completely eliminates this degradation because gradients from MTP training do not propagate into the NTP token paths.
Gated LoRA maintains original model accuracy compared to standard LoRA.
  • Sampler > Rank: Adding the lightweight MLP sampler head yields a significantly higher token acceptance rate than simply scaling up the LoRA rank. Remarkably, the model maintains strong MTP speedups even with a LoRA rank as low as 1.

Key Results

  • Massive Speedups in Structured Domains: When configured to predict 8 additional tokens, the model achieved a 5.35x speedup on coding (HumanEval) and a 5.22x speedup on math (GSM8k). These domains benefit most due to the high predictability of their future tokens.
  • Substantial Gains in Chat and Knowledge: General chat (AlpacaEval) and knowledge tasks saw speedups of roughly 2.4x to 2.5x.
  • Zero Quality Loss: Thanks to Gated LoRA, zero-shot accuracy on benchmarks like ARC-Challenge remained perfectly flat compared to the baseline model.
Figure 2: Speedup achieved after training gated LoRA and sampler head with supervised finetuning.

Limitations and Future Work

  • No Parameter Fusion: Because the LoRA layers are gated dynamically based on token type (mask vs. regular text), they cannot be permanently fused into the base model's weight matrices. This introduces a minor, though practically negligible, memory overhead during inference.
  • SFT-Limited Evaluation: The authors only evaluated this method during the supervised fine-tuning (SFT) stage. Exploring its impact during the initial large-scale pretraining phase remains an open and promising direction.
  • Diffusion Integration: The authors suggest that multi-token prediction lies on a spectrum between fully autoregressive and fully non-autoregressive models, noting that combining this approach with diffusion-based language models could yield further efficiency gains.

Summary updated

Abstract

Autoregressive language models are constrained by their inherently sequential nature, generating one token at a time. This paradigm limits inference speed and parallelism, especially during later stages of generation when the direction and semantics of text are relatively certain. In this work, we propose a novel framework that leverages the inherent knowledge of vanilla autoregressive language models about future tokens, combining techniques to realize this potential and enable simultaneous prediction of multiple subsequent tokens. Our approach introduces several key innovations: (1) a masked-input formulation where multiple future tokens are jointly predicted from a common prefix; (2) a gated LoRA formulation that preserves the original LLM's functionality, while equipping it for multi-token prediction; (3) a lightweight, learnable sampler module that generates coherent sequences from the predicted future tokens; (4) a set of auxiliary training losses, including a consistency loss, to enhance the coherence and accuracy of jointly generated tokens; and (5) a speculative generation strategy that expands tokens quadratically in the future while maintaining high fidelity. Our method achieves significant speedups through supervised fine-tuning on pretrained models. For example, it generates code and math nearly 5x faster, and improves general chat and knowledge tasks by almost 2.5x. These gains come without any loss in quality.
BibTeX
@misc{samragh2025your,
  author = {Mohammad Samragh and Arnav Kundu and David Harrison and Kumari Nishu and Devang Naik and Minsik Cho and Mehrdad Farajtabar},
  title = {{Your LLM Knows the Future: Uncovering Its Multi-Token Prediction Potential}},
  year = {2025},
  eprint = {2507.11851},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2507.11851},
}
Figure