TL;DR
Diffusion Forcing is a new training paradigm that bridges the gap between autoregressive next-token prediction and full-sequence diffusion by using independent, per-token noise levels. This allows models to maintain the flexible, variable-length generation of LLMs while gaining the stability and "guidable" planning capabilities of diffusion models, effectively solving the problem of compounding errors in long-horizon continuous generation.
What Problem Does This Paper Address?
In the world of sequence modeling, we usually have to choose between two flavors:
- Next-token prediction (Teacher Forcing): Think LLMs. They are great for variable-length sequences and efficient tree search, but they struggle with continuous data (like video) because small errors in one frame compound, causing the model to diverge quickly. They are also notoriously difficult to "guide" toward a specific future goal.
- Full-sequence diffusion: These models denoise an entire block of data at once. They are stable and easy to guide (e.g., "generate a path that reaches this coordinate"), but they are typically non-causal, have fixed horizons, and are computationally heavy for sequential decision-making.
The authors identify a missing middle ground: a model that is causal and flexible like an RNN, but stable and guidable like a diffusion model.
Methods
The core innovation is Diffusion Forcing (DF). Unlike standard diffusion, which applies the same noise level to an entire sequence, DF assigns an independent noise level $k_t$ to each token $x_t$.
Causal Diffusion Forcing (CDF)
The authors implement this using a causal architecture (specifically an RNN). The model's hidden state $z_t$ acts as a "belief" or summary of the past, even if that past is noisy.
- Training: The model is trained to denoise all tokens in a sequence simultaneously, but each token is corrupted by a random, independent amount of noise.
- Sampling: Because noise levels are independent, you can use unique schedules. For example, a "zig-zag" schedule allows the model to fully denoise the immediate next token while keeping the "far future" partially noisy (uncertain).
- Monte Carlo Guidance (MCG): This is a new capability where the model samples multiple potential futures to calculate a gradient that guides the current action toward a high-reward outcome.
Datasets
The authors evaluated Diffusion Forcing across four distinct domains:
- Video Generation: Minecraft gameplay and DMLab navigation videos.
- Planning: The D4RL benchmark (2D maze environments with sparse rewards).
- Robotics: A real-world Franka robot task involving swapping fruits in slots, which requires long-term memory.
- Time Series: Six standard GluonTS datasets, including Electricity, Traffic, and Wikipedia.
Key Insights and Ideas
- Noising as Partial Masking: The authors view noising as a generalization of masking. Zero noise is "unmasked," and full noise is "fully masked." This perspective allows DF to learn the joint distribution of all possible subsequences.
- Stabilizing the Rollout: One of the "aha" moments is using a small amount of noise during autoregressive generation. By telling the model that the previous frame is "slightly noisy" rather than "ground truth," the model becomes robust to its own small prediction errors, preventing divergence.
- The Bayesian Connection: DF effectively constructs an interpolation between the prior distribution (pure noise/no information) and the posterior distribution (clean data/full information) of a Bayes filter.
- Causal Uncertainty: By keeping the future "uncertain" (noisier) during sampling, the model avoids overcommitting to a single trajectory too early, which is critical for robust planning.
Key Results
- Infinite Video Rollout: While standard baselines diverged or became blurry within 72 frames, CDF successfully generated consistent video for over 1,000 frames (well past the training horizon).
- Superior Planning: In D4RL maze tasks, DF outperformed the state-of-the-art "Diffuser" model and standard RL baselines like CQL and IQL. The Monte Carlo Guidance (MCG) was shown to be a significant performance booster.
- Robotic Memory: In the fruit-swapping task, DF achieved an 80% success rate. A state-of-the-art "Diffusion Policy" failed because it lacked the latent memory required to remember the initial fruit positions.
- General Purpose: Despite not being specialized for time series, DF was competitive with or better than specialized SOTA models across most GluonTS benchmarks.
Limitations and Future Work
- Architecture Scaling: The current implementation relies on RNNs. While efficient for online control, the authors note that scaling to high-resolution, complex video will likely require moving to large Transformer backbones.
- Compute Overhead: Training with independent noise levels adds some temporal complexity. While the authors suggest "image pre-training" as a mitigation, the scaling behavior on internet-scale datasets remains an open question.
- Discrete Data: The paper focuses primarily on continuous tokens; extending the full benefits of DF to discrete text remains a promising future direction.
Summary updated