Planning with Diffusion for Flexible Behavior Synthesis

Michael Janner, Yilun Du, Joshua B. Tenenbaum, Sergey Levine

Introduction

Traditional model-based reinforcement learning (RL) typically separates world dynamics from planning. In these systems, a model predicts the next state based on the current one (autoregression), and a planner searches for the best sequence of actions. However, this approach suffers from two critical issues: adversarial exploitation, where planners find "holes" in the model that are physically impossible, and compounding errors, where small per-step prediction inaccuracies lead to significant "drift" over long horizons.

To address these challenges, we introduce Diffuser, a method that treats robot planning as a generative modeling problem. By using diffusion probabilistic models to generate and refine entire trajectories simultaneously, Diffuser avoids the pitfalls of step-by-step prediction and enables highly flexible behavior synthesis.

The Diffuser model refines a noisy set of states into a physically consistent trajectory through iterative denoising.

The Diffuser Framework

Diffuser represents a plan as a 2D array where dimensions correspond to time and state/action features. Instead of predicting the future one step at a time, the model starts with a "noisy" block of states and actions and iteratively refines them into a smooth, physically consistent trajectory.

The architecture utilizes a Temporal U-Net with 1D temporal convolutions. This fully convolutional design allows the model to generate plans of varying lengths and ensures global coherence. Because the model refines the entire path at once, it prioritizes the integrity of the whole trajectory over local, single-step accuracy, effectively neutralizing the problem of compounding errors.

Planning as Inference

One of the most powerful aspects of Diffuser is that planning is framed as a form of probabilistic inference. This allows for two key techniques borrowed from computer vision:

  1. Trajectory Inpainting: Similar to filling in missing pixels in an image, Diffuser can be constrained with a starting state and a target goal. It then "inpaints" the most likely physically consistent path between these two points. This allows for "anti-causal" reasoning, where the desired future constrains present decisions.
  2. Guided Sampling: To optimize for specific outcomes, a separate "guide" function (a small neural network) can nudge the denoising process toward trajectories that maximize a given reward signal.
Diffuser enables flexible planning through (a) denoising in complex environments, (b) stitching together prior data, (c) trajectory inpainting for goal-reaching, and (d) reward-guided sampling.

Evaluation and Results

Diffuser was evaluated across several challenging benchmarks, demonstrating superior performance in long-horizon tasks and test-time flexibility:

  • Long-Horizon Navigation: In the Large Maze2D task, which requires navigating complex environments with sparse rewards, Diffuser achieved a score of 123.0. This significantly outperformed traditional offline RL methods like CQL (12.5) and IQL (58.6).
  • Manipulation Flexibility: In block stacking tasks, Diffuser successfully handled novel "Rearrangement" goals (matching a block layout never seen during training) with a score of 58.9, while model-free baselines failed completely.
  • Temporal Compositionality: The model proved capable of "stitching" together snippets of behavior seen during training to solve new combinations of start and end points, a level of task agnosticism that standard RL agents often lack.

Conclusion and Future Work

Diffuser represents a shift toward treating planning as a holistic generative process. While the iterative nature of diffusion makes it computationally slower than single-pass policy networks, techniques like "warm-starting"—reusing previous plans to reduce denoising steps—offer a path toward real-time application. Future research will focus on accelerating the sampling process through methods like DDIM or distillation to bring this flexible planning framework to high-frequency robotic control.

Summary updated

Abstract

Model-based reinforcement learning methods often use learning only for the purpose of estimating an approximate dynamics model, offloading the rest of the decision-making work to classical trajectory optimizers. While conceptually simple, this combination has a number of empirical shortcomings, suggesting that learned models may not be well-suited to standard trajectory optimization. In this paper, we consider what it would look like to fold as much of the trajectory optimization pipeline as possible into the modeling problem, such that sampling from the model and planning with it become nearly identical. The core of our technical approach lies in a diffusion probabilistic model that plans by iteratively denoising trajectories. We show how classifier-guided sampling and image inpainting can be reinterpreted as coherent planning strategies, explore the unusual and useful properties of diffusion-based planning methods, and demonstrate the effectiveness of our framework in control settings that emphasize long-horizon decision-making and test-time flexibility.
BibTeX
@inproceedings{janner2022planning,
  author = {Michael Janner and Yilun Du and Joshua B. Tenenbaum and Sergey Levine},
  title = {{Planning with Diffusion for Flexible Behavior Synthesis}},
  booktitle = {International Conference on Machine Learning},
  year = {2022},
  url = {https://openreview.net/forum?id=NXyZfNISuG},
}
Figure