DIME:Diffusion-Based Maximum Entropy Reinforcement Learning

Onur Celik, Zechu Li, Denis Blessing, Ge Li, Daniel Palanicek, Jan Peters, Georgia Chalvatzaki, Gerhard Neumann

Abstract

DIME (Diffusion-Based Maximum Entropy Reinforcement Learning) bridges the gap between the high representational power of diffusion policies and the robust exploration framework of Maximum Entropy RL. By deriving a tractable lower bound for the marginal entropy of diffusion processes, DIME enables principled exploration in complex, high-dimensional control tasks. Our approach outperforms existing diffusion-based RL methods and matches or exceeds state-of-the-art Gaussian-based baselines across several challenging benchmarks.

Introduction

In Reinforcement Learning (RL), the Maximum Entropy (MaxEnt) framework—popularized by algorithms like Soft Actor-Critic (SAC)—is the standard for robust exploration. It encourages agents to maximize rewards while maintaining high action diversity (entropy). Traditionally, these policies are modeled as Gaussian distributions. While computationally efficient, Gaussians struggle to represent complex, multi-modal behaviors required for intricate tasks.

Diffusion models offer a far more expressive alternative, capable of representing nearly any distribution shape. However, calculating the entropy of a diffusion model is mathematically intractable. Previous attempts to use diffusion in RL often resorted to heuristic exploration, such as adding Gaussian noise to the final output, which undermines the benefits of using a sophisticated generative model. DIME addresses this by integrating diffusion policies directly into the MaxEnt objective through a mathematically rigorous entropy term.

Methodology

The core innovation of DIME is treating control as an inference problem. The authors derive a lower bound on the marginal entropy of the diffusion process, making the MaxEnt objective tractable for diffusion policies.

Entropy Lower Bound and Policy Iteration

Instead of calculating exact entropy, DIME utilizes the ratio between the forward (noising) and backward (denoising) processes in a discretized diffusion chain to create a tight lower bound. The authors prove that a policy iteration scheme using this bound monotonically improves the policy, eventually converging to an optimal diffusion policy.

Practical Implementation

The DIME algorithm incorporates several architectural improvements:

  • CrossQ Integration: By adopting the CrossQ architecture, DIME uses Batch Renormalization and eliminates target networks, significantly reducing computational overhead.
  • Distributional Q-Learning: The model predicts a distribution of possible rewards rather than a single expected value, enhancing performance in complex environments.
  • Autotuning: The algorithm automatically tunes the temperature parameter ($\alpha$), which balances exploration and exploitation, and the diffusion coefficient ($\beta$).

As shown in the figure above, the temperature parameter $\alpha$ directly influences the expressivity and entropy of the resulting policy. Lower values of $\alpha$ result in sharper, more multi-modal distributions, while higher values increase entropy, leading to broader and smoother distributions.

Experimental Evaluation

DIME was evaluated on 13 challenging continuous control benchmarks across three major suites:

  1. MuJoCo Gym: Standard locomotion tasks (e.g., Ant-v3, Humanoid-v3).
  2. DeepMind Control Suite (DMC): High-dimensional tasks involving "Dog" and "Humanoid" models.
  3. MyoSuite: Complex musculoskeletal simulations of a human hand (e.g., Pen Twirl, Key Turn), characterized by high-dimensional muscle-actuated action spaces.

Key Results and Insights

  • State-of-the-Art Performance: DIME consistently outperformed diffusion-based baselines such as QSM, DIPO, and Diff-QL.
  • Superiority over Gaussian Baselines: On the Humanoid-v3 task, DIME achieved significantly higher returns than all baselines. In the Dog-run task, it demonstrated faster convergence and higher final performance than top Gaussian-based methods like CrossQ and BRO.
  • Principled Exploration: Unlike methods that add noise post-hoc, DIME optimizes the entire diffusion process to be entropic. This allows for exploration patterns that are better suited to the environment's specific geometry.
  • Efficiency: Utilizing a lower Update-to-Data (UTD) ratio and the CrossQ architecture, DIME is significantly faster to train. For instance, it trains on the Humanoid task in approximately 4.5 hours, nearly twice as fast as the BRO baseline.

Conclusion and Future Work

DIME demonstrates that Gaussian assumptions in RL can be a bottleneck for complex robotics tasks. By providing a principled way to optimize the entropy of diffusion policies, DIME sets a new standard for expressive and efficient RL agents.

Future work includes exploring alternative diffusion types beyond the Ornstein-Uhlenbeck process and investigating different divergence measures (e.g., log-variance divergence) to further stabilize optimization. Additionally, scaling the architecture with Transformer-based critics could improve handling of temporal dependencies in RL.

Summary updated

BibTeX
@inproceedings{celik2025dime,
  author = {Onur Celik and Zechu Li and Denis Blessing and Ge Li and Daniel Palanicek and Jan Peters and Georgia Chalvatzaki and Gerhard Neumann},
  title = {{DIME:Diffusion-Based Maximum Entropy Reinforcement Learning}},
  booktitle = {International Conference on Machine Learning},
  year = {2025},
  url = {https://openreview.net/forum?id=Aw6dBR7Vxj},
}
Figure