TL;DR
Diffusion Forcing is a novel training paradigm that bridges the gap between autoregressive next-token prediction and full-sequence diffusion models. By training a causal model to denoise sequences where each token has an independent noise level, it enables stable, infinite-horizon video generation, flexible decision-making, and a powerful new planning technique called Monte Carlo Guidance.
What Problem Does This Paper Address?
In sequence modeling (like generating text, video, or robot actions), there are two dominant approaches, each with distinct strengths and fatal flaws:
- Next-Token Prediction (Teacher Forcing): Used by models like GPT, these predict the next token based on the past. They are great for generating variable-length sequences. However, on continuous data like video, small errors compound, causing the model to quickly diverge into garbage. Furthermore, you cannot easily "guide" the model toward a long-term goal because future tokens haven't been generated yet.
- Full-Sequence Diffusion: Used by models like Sora or Diffuser, these generate an entire sequence of fixed length all at once by gradually removing noise. They are great for continuous data and allow for "guidance" (steering the whole sequence toward a high reward). However, they are rigid (fixed horizons) and ignore causality—they denoise the immediate next step and the far future at the exact same rate, ignoring the fact that the far future is inherently more uncertain.
The authors tackle this gap: How can we build a model that has the flexible, causal nature of next-token prediction, but the stability and long-horizon guidance capabilities of full-sequence diffusion?
Methods
The authors introduce Diffusion Forcing (DF). The core conceptual leap is viewing diffusion noise as a form of partial masking. Zero noise means a token is fully visible; pure noise means it is fully masked.
Instead of denoising an entire sequence at the same noise level (standard diffusion) or fully masking all future tokens (standard next-token prediction), DF trains a causal neural network (like an RNN) on sequences where each token is assigned a random, independent noise level.
Because the model learns to denoise tokens regardless of the noise levels of surrounding tokens, you can design custom "noise schedules" at inference time. For example, you can fully denoise the immediate next frame while keeping frames 10 steps ahead heavily noised. This enables:
- Stabilized Autoregressive Rollout: By feeding slightly noised past frames into the model, it prevents compounding errors during long generations.
- Monte Carlo Guidance (MCG): Because DF can roll out variable futures, it can sample multiple possible future trajectories, calculate the expected reward, and use that gradient to guide the denoising of the current action.
Datasets
The authors evaluated DF across a remarkably diverse set of domains:
- Video Prediction: Minecraft Navigation and DeepMind Lab (DMLab) Navigation datasets.
- Planning / Offline RL: D4RL Maze2D benchmark (U-Maze, Medium, Large environments).
- Robotics: A custom real-world Franka robot dataset for a memory-dependent fruit-swapping task.
- Time Series Forecasting: GluonTS benchmarks (Exchange, Solar, Electricity, Traffic, Taxi, Wikipedia).
Key Insights and Ideas
- Causal Uncertainty is a Feature, Not a Bug: By keeping future tokens at a higher noise level than present tokens during sampling, DF naturally encodes the idea that the future is less certain. This prevents the model from prematurely committing to a rigid long-term plan, which is a common failure mode in standard diffusion planners.
- Noise as a Stabilizer: In autoregressive video generation, treating previously generated frames as "slightly noisy ground truth" acts as a regularizer. It forces the model to rely on its robust prior rather than over-indexing on tiny pixel errors, effectively solving the compounding error problem.
- One Model, Two Modes: DF can act as a fast, reactive policy (short horizon) or a slow, deliberate planner (long horizon) without changing the architecture. You simply change the sampling schedule matrix at inference time.
Key Results
- Infinite Video Rollout: DF stably rolled out continuous video for up to 2,000 frames despite only being trained on short sequences of 36-72 frames. Standard next-token and full-sequence diffusion baselines quickly diverged.
- Superior Planning: On the D4RL Maze2D benchmark, DF achieved a multi-task average score of 146.2, significantly outperforming the popular Diffuser model (129.4) and standard offline RL methods.
- Robust Real-World Robotics: On a physical robot task requiring memory (swapping hidden fruits), DF achieved an 80% success rate, whereas a standard diffusion policy completely failed (0%). When researchers threw visual distractors into the scene or occluded the camera, DF's success only dropped slightly to 76%, while baselines plummeted to 48%.
- Time Series: DF proved it is a strong general-purpose sequence model by achieving state-of-the-art or highly competitive CRPS (Continuous Ranked Probability Score) metrics across 6 standard forecasting datasets.
Limitations and Future Work
- Architecture constraints: The current implementation relies on Recurrent Neural Networks (RNNs). While RNNs are highly efficient for online decision-making and robotics, scaling DF to high-resolution video or internet-scale datasets will likely require adapting it to large Transformer architectures.
- Future directions: The authors note that applying DF to a non-causal Transformer—where causality is enforced purely by manipulating the noise levels of future tokens—is a highly promising direction for scaling up the paradigm.
Summary updated