Diffusion Guidance Is a Controllable Policy Improvement Operator

Kevin Frans, Seohong Park, Pieter Abbeel, Sergey Levine

Introduction

Reinforcement Learning (RL) has demonstrated remarkable success in complex decision-making tasks but remains notoriously difficult to scale due to training instability and extreme sensitivity to hyperparameters. Conversely, generative modeling techniques, such as diffusion and flow-matching, offer high stability and scalability. However, these models are typically confined to Imitation Learning (IL), where the agent's performance is capped by the quality of the training data.

The CFGRL (Classifier-Free Guidance for Reinforcement Learning) framework bridges this gap by reinterpreting diffusion guidance as a mathematically grounded policy improvement operator. By training a single generative model to represent both a "prior" behavior and an "optimal" behavior, researchers can use a guidance weight at test-time to "dial up" performance without retraining the model.

The Challenge of Policy Extraction

A central problem in offline RL is Policy Extraction: the ability to take a dataset of mixed-quality behaviors and extract a policy that outperforms the average or even the best trajectories in the data. Traditional methods like Advantage-Weighted Regression (AWR) often struggle with "gradient imbalance." In AWR, a small number of high-reward samples can dominate the training process, causing the model to ignore the majority of the dataset and potentially fail to generalize.

CFGRL addresses this by treating policy improvement as a product of two factors: a reference policy (the data distribution) and an "optimality" function. This approach allows the model to learn from the entire dataset while selectively emphasizing high-quality actions during inference.

Methods: The CFGRL Framework

The core of CFGRL lies in the application of Classifier-Free Guidance (CFG) to the policy distribution.

1. Product Policies as Improvement Operators

The authors prove that multiplying a reference policy by a function that increases with "advantage" (a measure of how much better an action is than the average) results in a distribution that is a guaranteed policy improvement. Mathematically, this aligns with KL-regularized RL objectives.

2. Classifier-Free Guidance (CFG)

Instead of training a separate optimality predictor, CFGRL trains a single diffusion or flow-matching network. During training, the optimality label is occasionally "dropped out" (replaced with a null token). This enables the network to learn both the unconditional distribution $\hat{\pi}(a|s)$ (the prior) and the conditional distribution $\hat{\pi}(a|s, o)$ (the optimal).

Comparison of unconditional, conditioned, and guided policies showing how guidance shifts the action distribution toward optimality.

3. Controllable Improvement at Test-Time

At inference, a guidance weight $w$ is used to combine these distributions:

  • $w=0$: Mimics the original dataset distribution.
  • $w=1$: Performs standard conditioning on optimality.
  • $w > 1$: "Amplifies" the optimality signal, pushing the agent toward higher-reward actions beyond what was seen in the training data.

The authors primarily utilize Flow-Matching as the backbone, a modern alternative to diffusion that offers faster and more stable training.

Experimental Evaluation

The authors evaluated CFGRL across several challenging benchmarks to test its robustness and scaling properties.

Datasets

  • ExORL: An offline RL benchmark featuring data from exploratory agents on tasks like Walker, Cheetah, and Jaco-arm.
  • OGBench: A suite for goal-conditioned RL, including state-based tasks (Pointmaze, Antmaze), visual tasks (pixel-based navigation), and hierarchical tasks requiring complex planning.

Key Results

  • Superior Offline Extraction: On ExORL, CFGRL consistently outperformed AWR. For instance, in the walker-walk task, CFGRL achieved a score of 608, significantly higher than AWR's 444.
  • Gains in Goal-Conditioned RL: In OGBench, CFGRL often doubled the success rate of standard Goal-Conditioned Behavioral Cloning (GCBC). In pointmaze-giant-navigate, CFGRL reached a 30% success rate, while the strongest baseline (Flow GCBC) achieved only 4%.
  • Reliable Scaling: Performance steadily increased with the guidance weight $w$ up to a certain threshold, confirming that guidance serves as a controllable operator for policy improvement.

Key Insights

  • Guidance as a "Knob": A significant finding is the mathematical equivalence between the guidance weight $w$ in diffusion and the inverse temperature $\beta$ in traditional KL-regularized RL. Unlike $\beta$, which must be fixed before training, $w$ can be tuned dynamically during inference.
  • Solving the AWR Bottleneck: While AWR effectively discards most of the dataset by focusing on high-weight samples, CFGRL learns from all data equally and uses the guidance mechanism to filter for quality during sampling.
  • "Free" Improvement for GCBC: Standard Goal-Conditioned Behavioral Cloning (GCBC) is shown to be a special case of CFGRL where $w=1$. Simply increasing $w$ at test-time yields a better policy without the need for value functions or critics in goal-conditioned settings.

Limitations and Future Work

Despite its strengths, CFGRL has limitations. For general offline RL, it still relies on a pre-trained value function to label data quality. Additionally, like other diffusion-based methods, inference is slower due to the iterative sampling process. Pushing the guidance weight $w$ too high can also lead to "distribution shift," where the model generates unrealistic actions. Future research may focus on integrating CFGRL with online RL to further enhance extrapolation capabilities.

Conclusion

CFGRL provides a powerful, stable, and controllable framework for policy improvement in RL. By leveraging the strengths of generative modeling and classifier-free guidance, it offers a promising path toward more scalable and robust autonomous agents.

Summary updated

Abstract

At the core of reinforcement learning is the idea of learning beyond the performance in the data. However, scaling such systems has proven notoriously tricky. In contrast, techniques from generative modeling have shown to be remarkably scalable and are simple to train. In this work, we combine these strengths, by deriving a direct relation between policy improvement and guidance of diffusion models. The resulting framework, CFGRL, is a policy improvement operator that is trained with the simplicity of supervised learning, yet is more effective than typically-used weighted policy extraction strategies. On offline RL tasks, we observe a reliable trend---increased guidance weighting leads to increased performance. Additionally, the CFGRL framework can be adapted to "directly'' extract policies from offline data *without* running a full end-to-end RL algorithm, allowing us to generalize simple supervised methods (e.g. goal-conditioned behavior cloning) to further prioritize optimality, gaining performance across the board without additional cost.
BibTeX
@misc{frans2025diffusion,
  author = {Kevin Frans and Seohong Park and Pieter Abbeel and Sergey Levine},
  title = {{Diffusion Guidance Is a Controllable Policy Improvement Operator}},
  year = {2025},
  eprint = {2505.23458},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2505.23458},
}
Figure