Diffusion Forcing: Next-token Prediction Meets Full-Sequence Diffusion

Boyuan Chen, Diego Marti Monso, Yilun Du, Max Simchowitz, Russ Tedrake, Vincent Sitzmann

TL;DR

Diffusion Forcing is a new training paradigm that bridges the gap between autoregressive next-token prediction and full-sequence diffusion by using independent, per-token noise levels. This allows models to maintain the flexible, variable-length generation of LLMs while gaining the stability and "guidable" planning capabilities of diffusion models, effectively solving the problem of compounding errors in long-horizon continuous generation.

What Problem Does This Paper Address?

In the world of sequence modeling, we usually have to choose between two flavors:

  1. Next-token prediction (Teacher Forcing): Think LLMs. They are great for variable-length sequences and efficient tree search, but they struggle with continuous data (like video) because small errors in one frame compound, causing the model to diverge quickly. They are also notoriously difficult to "guide" toward a specific future goal.
  2. Full-sequence diffusion: These models denoise an entire block of data at once. They are stable and easy to guide (e.g., "generate a path that reaches this coordinate"), but they are typically non-causal, have fixed horizons, and are computationally heavy for sequential decision-making.

The authors identify a missing middle ground: a model that is causal and flexible like an RNN, but stable and guidable like a diffusion model.

Methods

The core innovation is Diffusion Forcing (DF). Unlike standard diffusion, which applies the same noise level to an entire sequence, DF assigns an independent noise level $k_t$ to each token $x_t$.

Figure 2: Diffusion Forcing interleaves the time axis and the noise axis, allowing for flexible-length sequences where each frame has a different noise level.

Causal Diffusion Forcing (CDF)

The authors implement this using a causal architecture (specifically an RNN). The model's hidden state $z_t$ acts as a "belief" or summary of the past, even if that past is noisy.

  • Training: The model is trained to denoise all tokens in a sequence simultaneously, but each token is corrupted by a random, independent amount of noise.
  • Sampling: Because noise levels are independent, you can use unique schedules. For example, a "zig-zag" schedule allows the model to fully denoise the immediate next token while keeping the "far future" partially noisy (uncertain).
  • Monte Carlo Guidance (MCG): This is a new capability where the model samples multiple potential futures to calculate a gradient that guides the current action toward a high-reward outcome.
Figure 1: Diffusion Forcing capabilities. Today, different applications such as language modeling [6], planning [37], or video generation [32, 70] rely on either auto-regressive next-token prediction or full-sequence diffusion, according to their respective unique capabilities. The proposed Diffusion Forcing is a novel sequence generative model that enjoys key strengths of both model types.

Datasets

The authors evaluated Diffusion Forcing across four distinct domains:

  1. Video Generation: Minecraft gameplay and DMLab navigation videos.
  2. Planning: The D4RL benchmark (2D maze environments with sparse rewards).
  3. Robotics: A real-world Franka robot task involving swapping fruits in slots, which requires long-term memory.
  4. Time Series: Six standard GluonTS datasets, including Electricity, Traffic, and Wikipedia.

Key Insights and Ideas

  • Noising as Partial Masking: The authors view noising as a generalization of masking. Zero noise is "unmasked," and full noise is "fully masked." This perspective allows DF to learn the joint distribution of all possible subsequences.
  • Stabilizing the Rollout: One of the "aha" moments is using a small amount of noise during autoregressive generation. By telling the model that the previous frame is "slightly noisy" rather than "ground truth," the model becomes robust to its own small prediction errors, preventing divergence.
  • The Bayesian Connection: DF effectively constructs an interpolation between the prior distribution (pure noise/no information) and the posterior distribution (clean data/full information) of a Bayes filter.
  • Causal Uncertainty: By keeping the future "uncertain" (noisier) during sampling, the model avoids overcommitting to a single trajectory too early, which is critical for robust planning.
Figure 5: By controlling the noise level $k$, the model can treat observations as ground truth, noisy, or as predictions to be generated.

Key Results

  • Infinite Video Rollout: While standard baselines diverged or became blurry within 72 frames, CDF successfully generated consistent video for over 1,000 frames (well past the training horizon).
  • Superior Planning: In D4RL maze tasks, DF outperformed the state-of-the-art "Diffuser" model and standard RL baselines like CQL and IQL. The Monte Carlo Guidance (MCG) was shown to be a significant performance booster.
  • Robotic Memory: In the fruit-swapping task, DF achieved an 80% success rate. A state-of-the-art "Diffusion Policy" failed because it lacked the latent memory required to remember the initial fruit positions.
  • General Purpose: Despite not being specialized for time series, DF was competitive with or better than specialized SOTA models across most GluonTS benchmarks.
Figure 4: The robot task requires memory of the initial state to succeed; DF's latent state successfully captures this context.

Limitations and Future Work

  • Architecture Scaling: The current implementation relies on RNNs. While efficient for online control, the authors note that scaling to high-resolution, complex video will likely require moving to large Transformer backbones.
  • Compute Overhead: Training with independent noise levels adds some temporal complexity. While the authors suggest "image pre-training" as a mitigation, the scaling behavior on internet-scale datasets remains an open question.
  • Discrete Data: The paper focuses primarily on continuous tokens; extending the full benefits of DF to discrete text remains a promising future direction.

Summary updated

Abstract

This paper presents Diffusion Forcing, a new training paradigm where a diffusion model is trained to denoise a set of tokens with independent per-token noise levels. We apply Diffusion Forcing to sequence generative modeling by training a causal next-token prediction model to generate one or several future tokens without fully diffusing past ones. Our approach is shown to combine the strengths of next-token prediction models, such as variable-length generation, with the strengths of full-sequence diffusion models, such as the ability to guide sampling to desirable trajectories. Our method offers a range of additional capabilities, such as (1) rolling-out sequences of continuous tokens, such as video, with lengths past the training horizon, where baselines diverge and (2) new sampling and guiding schemes that uniquely profit from Diffusion Forcing's variable-horizon and causal architecture, and which lead to marked performance gains in decision-making and planning tasks. In addition to its empirical success, our method is proven to optimize a variational lower bound on the likelihoods of all subsequences of tokens drawn from the true joint distribution. Project website: https://boyuan.space/diffusion-forcing
BibTeX
@inproceedings{chen2024diffusion,
  author = {Boyuan Chen and Diego Marti Monso and Yilun Du and Max Simchowitz and Russ Tedrake and Vincent Sitzmann},
  title = {{Diffusion Forcing: Next-token Prediction Meets Full-Sequence Diffusion}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2024},
  url = {https://openreview.net/forum?id=yDo1ynArjj},
}
Figure