Diffusion Guidance Is a Controllable Policy Improvement Operator

Kevin Frans, Seohong Park, Pieter Abbeel, Sergey LevineSubmission to ICLR 2026 · Rejected

TL;DR

This paper demonstrates that classifier-free guidance (CFG) in diffusion and flow-matching models functions mathematically and practically as a policy improvement operator in reinforcement learning. By framing an improved policy as the product of a reference policy and an optimality factor, the authors introduce CFGRL—a method that provides controllable policy improvement at test time simply by tuning the guidance scale, completely avoiding retraining or fragile advantage-weighted regression.

What Problem Does This Paper Address?

In offline reinforcement learning (RL), agents must extract high-performing policies from fixed datasets collected by suboptimal or exploratory behavior. Traditional methods typically rely on Advantage-Weighted Regression (AWR), where actions are weighted by exponentiated advantages. However, AWR suffers from serious practical flaws:

  1. Gradient imbalance: Exponential weights can cause a few outlier transitions to dominate training batches, effectively discarding most data.
  2. Fixed conservatism: The trade-off between maximizing reward and staying close to the behavior data distribution (the temperature hyperparameter $\beta$) is hard-coded during training; tuning it requires full retraining.

Meanwhile, expressive generative policies (such as diffusion and flow matching) are stable and scalable for imitation learning, but standard implementations lack a clean, controllable mechanism to optimize beyond data-level performance without complex critics or backpropagation through time.

Methods

The authors build on the control-as-inference framework to represent an improved policy as a product distribution: $$\pi(a \mid s) \propto \hat{\pi}(a \mid s) \, f(A(s, a))$$ where $\hat{\pi}$ is a reference policy and $f(A(s, a))$ is a non-negative, non-decreasing function of the advantage function $A(s, a)$.

  • Guiding Action Generation: Instead of explicitly modeling the unnormalized score of $f$, the authors formulate optimality as a binary variable $o \in \{0, 1\}$ (e.g., $o = 1$ if $A(s, a) \ge 0$). Applying Bayes’ rule yields the score: $$\nabla_a \log \pi(a \mid s) = \nabla_a \log \hat{\pi}(a \mid s) + w \Big(\nabla_a \log \hat{\pi}(a \mid s, o) - \nabla_a \log \hat{\pi}(a \mid s)\Big)$$ This directly mirrors Classifier-Free Guidance (CFG), where the guidance weight $w$ controls the degree of policy improvement dynamically at inference time.

  • Training with Flow Matching: A single velocity network $v_\theta(a_t, t, s, o)$ is trained with standard supervised flow-matching loss (using 10% dropout on $o$ to learn the unconditional reference $\hat{\pi}$). Every sample in the batch is weighted equally.

  • Value-Free Goal-Conditioned BC (GCBC): The authors extend this insight to goal-conditioned RL. By noting that standard GCBC implicitly defines a product policy with $w = 1$, they show that applying guidance toward the goal ($w > 1$) relative to an unconditional prior yields provable policy improvement without training an explicit value function or critic.

Datasets

The authors evaluate CFGRL on two benchmark suites:

  • ExORL: 9 continuous-control robotic tasks (e.g., Walker, Cheetah, Quadruped, Jaco arm) using exploratory offline datasets generated via Random Network Distillation (RND).
  • OGBench: 9 single-task environments for offline RL and 24 goal-conditioned tasks (17 state-based, 7 pixel-based) covering complex navigation, multi-stage manipulation, whole-body humanoid locomotion, and combinatorial puzzles.

Key Insights and Ideas

  • Diffusion Guidance Provably Improves Policies: Using Chebyshev’s sum inequality, the authors prove that compounding a reference distribution with an attenuated, monotonically increasing function of advantage ($w_2 > w_1 \ge 0$) strictly guarantees higher expected advantage.
  • Test-Time Regularization Tuning: In AWR, sweeping the temperature requires retraining from scratch. In CFGRL, the policy is trained once, and the trade-off between conservatism and reward maximization is dialed in via sampling-time guidance weight $w$.
  • "Free" Improvement over GCBC: Contrasting a goal-conditioned policy against an unconditional policy during sampling ($w > 1$) yields an immediate boost in goal achievement over standard behavioral cloning without requiring a value network.

Key Results

  • Outperforming AWR in Offline RL: Across ExORL and OGBench tasks (paired with value functions learned via Implicit Q-Learning, IQL), CFGRL consistently outperformed AWR.
  • On ExORL walker-stand, CFGRL achieved 782 ± 8 vs. AWR's 603 ± 8.
  • On cheetah-run-backward, CFGRL scored 262 ± 26 vs. AWR's 146 ± 8.
  • On OGBench pointmaze-teleport, CFGRL reached 57 ± 7 vs. AWR's 3 ± 7.
  • As guidance scale $w$ increased, CFGRL continued to scale in performance where AWR’s inverse temperature $1/\beta$ saturated or degraded.
  • Substantial Gains in Goal-Conditioned Settings:
  • On difficult tasks like pointmaze-giant, CFGRL attained a 30 ± 10% success rate compared to 4 ± 2% for Flow GCBC.
  • On the vision-based visual-cube-single manipulation task, CFGRL achieved 37 ± 9% vs. 13 ± 7% for Flow GCBC.
  • In hierarchical settings (HCFGRL), humanoid maze navigation jumped from 23 ± 4% (Flow HGCBC) to 64 ± 10%.

Limitations and Future Work

  • Not an End-to-End RL Algorithm: CFGRL acts as a policy extraction or improvement operator; in standard offline RL, it still requires a pre-trained critic (such as IQL).
  • Extrapolation Limits: Pushing $w$ too high eventually degrades performance because the generated actions push too far out of the data distribution, causing distribution shift.
  • Evaluation Speed: Like all diffusion- and flow-based models, CFGRL requires multi-step numerical integration at inference time (e.g., 16–32 steps), which is slower than single-forward-pass feedforward actors.

Summary updated

Abstract

At the core of reinforcement learning is the idea of learning beyond the performance in the data. However, scaling such systems has proven notoriously tricky. In contrast, techniques from generative modeling have shown to be remarkably scalable and are simple to train. In this work, we combine these strengths, by deriving a direct relation between policy improvement and guidance of diffusion models. The resulting framework, CFGRL, is a policy improvement operator that is trained with the simplicity of supervised learning, yet is more effective than typically-used weighted policy extraction strategies. On offline RL tasks, we observe a reliable trend---increased guidance weighting leads to increased performance. Additionally, the CFGRL framework can be adapted to "directly'' extract policies from offline data *without* running a full end-to-end RL algorithm, allowing us to generalize simple supervised methods (e.g. goal-conditioned behavior cloning) to further prioritize optimality, gaining performance across the board without additional cost.
BibTeX
@misc{frans2025diffusion,
  author = {Kevin Frans and Seohong Park and Pieter Abbeel and Sergey Levine},
  title = {{Diffusion Guidance Is a Controllable Policy Improvement Operator}},
  year = {2025},
  eprint = {2505.23458},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2505.23458},
}
Figure