Diffusion Forcing: Next-token Prediction Meets Full-Sequence Diffusion

Boyuan Chen, Diego Marti Monso, Yilun Du, Max Simchowitz, Russ Tedrake, Vincent Sitzmann

Abstract

This paper presents Diffusion Forcing, a new training paradigm where a diffusion model is trained to denoise a set of tokens with independent per-token noise levels. We apply Diffusion Forcing to sequence generative modeling by training a causal next-token prediction model to generate one or several future tokens without fully diffusing past ones. Our approach is shown to combine the strengths of next-token prediction models, such as variable-length generation, with the strengths of full-sequence diffusion models, such as the ability to guide sampling to desirable trajectories. Our method offers a range of additional capabilities, such as (1) rolling-out sequences of continuous tokens, such as video, with lengths past the training horizon, where baselines diverge and (2) new sampling and guiding schemes that uniquely profit from Diffusion Forcing's variable-horizon and causal architecture, and which lead to marked performance gains in decision-making and planning tasks. In addition to its empirical success, our method is proven to optimize a variational lower bound on the likelihoods of all subsequences of tokens drawn from the true joint distribution.

Introduction

Sequence modeling has traditionally been split between two paradigms: autoregressive next-token prediction (Teacher Forcing) and full-sequence diffusion. While Teacher Forcing, the backbone of modern Large Language Models (LLMs), excels at handling variable-length sequences and respecting causality, it often fails on continuous data like video due to compounding errors and "drift." Conversely, full-sequence diffusion models offer stability and ease of goal-directed guidance but are typically non-causal and restricted to fixed sequence lengths.

Diffusion Forcing (DF) bridges this gap, creating a system that is causal like a language model yet stable and "guidable" like a diffusion model—enabling applications ranging from infinite video rollouts to high-performance robotic planning.

Methodology

The core innovation of Diffusion Forcing is training a model to denoise sequences where each token is assigned an independent, random noise level. This contrasts with standard diffusion, which noises an entire sequence uniformly.

Method Overview: Diffusion Forcing unifies next-token prediction and full-sequence diffusion by training on independent noise levels per token.

Key technical components include:

  • Causal Architecture: Utilizing structures like Recurrent Neural Networks (RNNs) ensures that predictions for a given token depend only on the noisy versions of preceding tokens.
  • Noise as Partial Masking: Noising is treated as a spectrum of masking, where zero noise represents full visibility and maximum noise represents a completely hidden token. This "forces" the model to learn information recovery from various combinations of partially masked inputs.
  • Monte Carlo Guidance (MCG): The causal nature allows the model to simulate multiple potential futures. MCG leverages these rollouts to guide the current step toward high-reward outcomes with lower variance than traditional diffusion guidance.

Experimental Evaluation

The authors evaluated Diffusion Forcing across four diverse domains:

  • Video Generation: Minecraft gameplay and DMLab navigation videos.
  • Motion Planning: 2D maze environments with sparse rewards (D4RL benchmark).
  • Robotics: A real-world Franka robot task requiring memory of initial states to swap fruit positions.
  • Time Series Forecasting: Six standard datasets from the GluonTS library (Exchange, Solar, Electricity, Traffic, Taxi, and Wikipedia).

Key Insights and Capabilities

Diffusion Forcing introduces several unique advantages over existing methods. By training on noisy inputs, the model becomes robust to its own imperfect predictions during sampling, enabling stable "infinite" rollouts.

Comparison of capabilities across Teacher Forcing, Full-Sequence Diffusion, and Diffusion Forcing.
  • Causal Uncertainty: The authors introduce a "Zig-Zag" sampling schedule that allows the model to denoise the near future quickly while keeping the distant future noisy, reflecting the natural increase in uncertainty over time.
  • Flexible Guidance: The independent noise levels allow for changing the planning horizon or guidance strength at test time without retraining the model weights.
  • Robustness to Corruption: The model can handle corrupted or missing observations. By treating a specific frame as high-noise, the model ignores corrupted visual data and relies on its internal latent state to continue the task.

Results

  • Stable Video: While standard baselines diverged after a few dozen frames, Diffusion Forcing successfully rolled out consistent video for over 1,000 frames, well beyond its training horizon.
  • Superior Planning: In Maze2D benchmarks, DF significantly outperformed "Diffuser" (a state-of-the-art diffusion planner), nearly doubling performance in complex multi-goal scenarios.
  • Stateful Robotics: DF achieved an 80% success rate in memory-dependent tasks where standard non-recurrent Diffusion Policies failed completely.

Conclusion

Diffusion Forcing represents a significant step toward unifying autoregressive and diffusion-based sequence modeling. While current implementations rely on RNNs, the authors note that scaling to high-resolution video will likely require Transformer-based backbones with causal masking. The method's ability to combine causality with the stability of diffusion opens new avenues for robust, long-horizon sequence generation and planning.

Summary updated

Abstract

This paper presents Diffusion Forcing, a new training paradigm where a diffusion model is trained to denoise a set of tokens with independent per-token noise levels. We apply Diffusion Forcing to sequence generative modeling by training a causal next-token prediction model to generate one or several future tokens without fully diffusing past ones. Our approach is shown to combine the strengths of next-token prediction models, such as variable-length generation, with the strengths of full-sequence diffusion models, such as the ability to guide sampling to desirable trajectories. Our method offers a range of additional capabilities, such as (1) rolling-out sequences of continuous tokens, such as video, with lengths past the training horizon, where baselines diverge and (2) new sampling and guiding schemes that uniquely profit from Diffusion Forcing's variable-horizon and causal architecture, and which lead to marked performance gains in decision-making and planning tasks. In addition to its empirical success, our method is proven to optimize a variational lower bound on the likelihoods of all subsequences of tokens drawn from the true joint distribution. Project website: https://boyuan.space/diffusion-forcing
BibTeX
@inproceedings{chen2024diffusion,
  author = {Boyuan Chen and Diego Marti Monso and Yilun Du and Max Simchowitz and Russ Tedrake and Vincent Sitzmann},
  title = {{Diffusion Forcing: Next-token Prediction Meets Full-Sequence Diffusion}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2024},
  url = {https://openreview.net/forum?id=yDo1ynArjj},
}
Figure