TL;DR
Autoregressive language models are notoriously slow because they generate text one token at a time. This paper introduces a novel framework that unlocks an LLM's latent ability to predict multiple future tokens simultaneously using masked inputs and a unique "Gated LoRA" mechanism. The result is a massive inference speedup—up to 5x in code and math generation—with absolutely zero loss in model quality.
What Problem Does This Paper Address?
Standard Large Language Models (LLMs) operate autoregressively: to generate a sentence, they predict the next token, append it to the prompt, and repeat. While highly effective, this sequential bottleneck makes inference computationally expensive and slow.
To accelerate generation, researchers have turned to Multi-Token Prediction (MTP) and speculative decoding. Speculative decoding typically relies on a separate, smaller "draft" model to guess future tokens, which adds architectural complexity. Direct MTP trains a single model to output multiple tokens at once, but fine-tuning an existing model for MTP often degrades its core Next-Token Prediction (NTP) accuracy—a phenomenon akin to catastrophic forgetting.
The authors of this paper ask a fundamental question: Can we train a single model to reliably predict multiple future tokens without sacrificing its original generation quality?
Methods
The authors propose a clean, modular architecture to turn a standard autoregressive LLM into an efficient multi-token predictor:
- Masked-Input Formulation: Instead of adding heavy, complex prediction heads, the authors append $k$ special
<mask>tokens to the end of a prompt. The model uses its full depth and capacity to fill in these masks with future tokens. - Gated LoRA: To protect the base model's original performance, the authors introduce a binary gating mechanism for low-rank adapters (LoRA). The LoRA path is activated ($I(t)=1$) only when processing
<mask>tokens (MTP) and completely bypassed ($I(t)=0$) for regular text tokens (NTP). This ensures the original autoregressive capabilities remain completely untouched. - Lightweight Sampler Module: To ensure the predicted future tokens form a coherent sequence, they feed the model's hidden states into a lightweight, two-layer Multi-Layer Perceptron (MLP) sampler. This sampler conditions each predicted mask token on both the current context and the immediately preceding sampled token.
- Latent Consistency Loss (LCM): This auxiliary self-distillation loss forces the hidden representations of early mask tokens to align closely with the actual future hidden states of the autoregressive model, improving prediction accuracy.
- Quadratic Decoding: A speculative decoding strategy that interleaves mask tokens within speculative tokens in a tree-like structure. This ensures that even if a speculative token chain breaks early during verification, a steady stream of tokens is still available for verification at each step.
Datasets
The framework was evaluated by fine-tuning a Tulu3-8B SFT model using the Tulu3 dataset, which contains nearly 1 million examples spanning math, code, conversation, and scientific benchmarks.
To measure speedups and verify that no quality degradation occurred, the authors tested the model across several prominent benchmarks:
- Knowledge: MMLU, PopQA, TruthfulQA
- Math & Coding: GSM8k, HumanEval
- Chat: AlpacaEval, IFEval
- Safety: XSTest, HarmBench, WildGuardTest, and others.
Key Insights and Ideas
- The Future is Already There: Intriguingly, pretrained LLMs already implicitly know the future. When given placeholder tokens, a vanilla model ranks the correct future tokens within its top-200 logits. Fine-tuning with mask tokens simply coaxes this latent knowledge into the top-10.
- Standard LoRA Degrades Capabilities: Fine-tuning an SFT model with standard LoRA causes a sharp drop in downstream task accuracy (e.g., ARC-Challenge). Gated LoRA completely eliminates this degradation because gradients from MTP training do not propagate into the NTP token paths.
- Sampler > Rank: Adding the lightweight MLP sampler head yields a significantly higher token acceptance rate than simply scaling up the LoRA rank. Remarkably, the model maintains strong MTP speedups even with a LoRA rank as low as 1.
Key Results
- Massive Speedups in Structured Domains: When configured to predict 8 additional tokens, the model achieved a 5.35x speedup on coding (HumanEval) and a 5.22x speedup on math (GSM8k). These domains benefit most due to the high predictability of their future tokens.
- Substantial Gains in Chat and Knowledge: General chat (AlpacaEval) and knowledge tasks saw speedups of roughly 2.4x to 2.5x.
- Zero Quality Loss: Thanks to Gated LoRA, zero-shot accuracy on benchmarks like ARC-Challenge remained perfectly flat compared to the baseline model.
Limitations and Future Work
- No Parameter Fusion: Because the LoRA layers are gated dynamically based on token type (mask vs. regular text), they cannot be permanently fused into the base model's weight matrices. This introduces a minor, though practically negligible, memory overhead during inference.
- SFT-Limited Evaluation: The authors only evaluated this method during the supervised fine-tuning (SFT) stage. Exploring its impact during the initial large-scale pretraining phase remains an open and promising direction.
- Diffusion Integration: The authors suggest that multi-token prediction lies on a spectrum between fully autoregressive and fully non-autoregressive models, noting that combining this approach with diffusion-based language models could yield further efficiency gains.
Summary updated