In traditional reinforcement learning (RL), we typically represent an agent's policy as a simple probability distribution—most often a unimodal Gaussian. While mathematically convenient, this "bell curve" assumption creates a fundamental bottleneck: it assumes there is only one "correct" way to act in any given state.
In the real world, this is rarely true. If a robot encounters an obstacle, it can go left or right. Both are optimal. However, a Gaussian policy forced to choose between these two modes will often settle on the average of the two—driving straight into the obstacle. Researchers are now overcoming this "averaging" problem by replacing simple distributions with generative diffusion and flow-based models, allowing agents to represent complex, multi-modal behaviors with unprecedented expressivity.
The Multi-Modality Problem
The core motivation for this shift is the failure of standard policies to capture the diversity of expert data. As noted in Diffusion Models for Reinforcement Learning: A Survey, traditional neural networks tend to over-smooth diverse expert actions. This is particularly damaging in offline RL, where the training data might contain multiple successful strategies for the same task.
Zhendong Wang and colleagues addressed this in Diffusion Policies as an Expressive Policy Class for Offline Reinforcement Learning by introducing Diffusion-QL. Instead of outputting the parameters of a distribution, the policy is defined as a conditional reverse diffusion process. It starts with pure noise and iteratively refines it into an action, conditioned on the current state. This allows the policy to "shape" itself around multiple peaks in the action space, accurately reflecting the multi-modal nature of human or mixed-agent datasets.
Linking Diffusion to Rewards
While diffusion models are excellent at mimicking data (behavior cloning), RL requires optimizing for rewards. A key breakthrough in this direction is Q-Score Matching (QSM), developed by Michael Psenka, Pieter Abbeel, and others in Learning a Diffusion Model Policy from Rewards via Q-Score Matching.
QSM provides the mathematical bridge: it links the "score" (the gradient of the log-density) of a diffusion model directly to the gradient of a Q-function. This allows the agent to learn complex behaviors more stably than previous methods by ensuring the diffusion process naturally gravitates toward high-reward regions of the action space. Similarly, Shutong Ding and the authors of QVPO demonstrated that these expressive policies can be successfully trained in online settings, achieving superior sample efficiency by capturing the full distribution of optimal actions.
From Diffusion to Flow Matching
Despite their expressivity, diffusion models suffer from a major drawback: they are slow. Iteratively denoising an action over dozens of steps is computationally expensive for real-time robotics.
To solve this, researchers are pivoting toward Flow Matching. In Flow Q-Learning, Seohong Park and Sergey Levine utilize flow-matching architectures to learn a continuous transformation from a simple base distribution to the target action distribution. This approach maintains the multi-modal benefits of diffusion but allows for much faster, often one-step, inference.
Other efficiency-focused methods include:
- Consistency Policies: Zihan Ding and Chi Jin introduced Consistency Models as a Rich and Efficient Policy Class, which distill the multi-step diffusion process into a single-step mapping, drastically reducing inference latency for online interaction.
- Flow Policy Mirror Descent (FPMD): Bo Dai and colleagues developed FPMD, which leverages the fact that RL policies eventually converge to low-variance distributions to enable state-of-the-art performance with 1-step sampling.
Real-World Impact: Dexterity and Safety
The practical benefits of these expressive policies are most visible in complex physical tasks. In DexDiff, Chengzhong Ma and team used diffusion policies to enable robots to manipulate "ungraspable" objects by using the environment (like table edges). Because the task has many valid solutions, the diffusion model's ability to represent diverse successful strategies—rather than an average "middle" path—was critical for success.
Furthermore, generative policies are proving essential for Safe RL. Yinan Zheng and the authors of FISOR showed that diffusion models can effectively model the boundaries of safe behavior. By using a feasibility-guided diffusion process, they can extract policies that maximize rewards while strictly adhering to safety constraints, avoiding the "unsafe averages" that plague simpler models.
By moving beyond the Gaussian bell curve, this research direction is transforming RL agents from rigid optimizers into flexible, generative actors capable of mastering the messy, multi-modal reality of the physical world.