Maximum Entropy Inverse Reinforcement Learning of Diffusion Models with Energy-Based Models

Sangwoong Yoon, Himchan Hwang, Dohyun Kwon, Yung-Kyun Noh, Frank C. Park

Abstract

Standard diffusion models often suffer from distribution shifts and blurry outputs when sampled in a small number of steps. This paper introduces DxMI (Diffusion by Maximum Entropy IRL), a framework that reinterprets diffusion models as reinforcement learning policies and energy-based models (EBMs) as reward functions. By employing a novel dynamic programming approach (DxDP) to bypass memory-intensive backpropagation through time, the authors achieve high-quality image generation in as few as 4 to 10 steps and set new benchmarks in anomaly detection.

Introduction: Beyond Behavioral Cloning

Standard diffusion models are typically trained using a method analogous to behavioral cloning: they learn to mimic a predefined "expert" trajectory that gradually transforms noise into data over 1,000+ steps. However, a significant "distribution shift" occurs when these models are accelerated for real-time generation (e.g., 10 steps). The model encounters intermediate states it never saw during its fine-grained training, leading to distorted samples.

While distillation and adversarial training (GANs) can mitigate this, they often require pre-trained teacher models or suffer from training instability. This work argues that Inverse Reinforcement Learning (IRL) provides a superior alternative: instead of merely mimicking steps, the model learns the underlying "reward" (data density) and optimizes its path to maximize that reward.

The DxMI Framework

The authors propose a minimax game between two components:

  1. Energy-Based Model (EBM): Acts as the reward function, estimating the log-probability density of the training data.
  2. Diffusion Model: Acts as the policy, attempting to generate samples that receive high rewards from the EBM while maximizing its own entropy to ensure diversity.
The DxMI framework: Diffusion models act as policies $\pi(x)$ and EBMs provide rewards based on data density $q(x)$.

Efficient Optimization via DxDP

Updating a diffusion model via RL is traditionally computationally prohibitive due to the need to backpropagate gradients through every timestep. To solve this, the authors introduce DxDP (Diffusion by Dynamic Programming). By transforming the problem into an optimal control formulation, DxDP uses value functions (learned via the Bellman residual) to update the model locally at each timestep. This removes the need to backpropagate through the entire chain, significantly reducing GPU memory usage and stabilizing gradients.

Key Insights: Entropy and Stability

A critical finding of this research is that maximizing entropy does more than just ensure sample diversity; it stabilizes the training of the EBM itself. By incorporating entropy regularization, the EBM can be trained without the expensive Markov Chain Monte Carlo (MCMC) sampling that typically bottlenecks EBM research.

Impact of entropy regularization ($\tau$) on the learned energy landscape $E_\theta(x)$ compared to the true distribution.

As shown in the synthetic 2D experiments, the temperature parameter $\tau$ (representing entropy) allows the model to capture the global structure of the data distribution more effectively than standard EBM training.

Experimental Results

The researchers evaluated DxMI across image generation and anomaly detection benchmarks:

  • High-Quality Few-Step Generation: On CIFAR-10, DxMI achieved an FID of 3.19 in just 10 steps, significantly outperforming standard DDIM (13.36) and specialized policy gradient methods. Qualitative results on ImageNet 64×64 and LSUN Bedroom 256×256 showed superior structural integrity compared to Consistency Models.
  • State-of-the-Art Anomaly Detection: On the MVTec-AD benchmark, DxMI achieved a mean Area Under the Curve (AUC) of 97.0% for detection and 97.1% for localization, surpassing previous state-of-the-art models like UniAD.
  • Computational Efficiency: Training on CIFAR-10 requires less than 4 hours on four A100 GPUs, making the framework significantly more accessible than large-scale distillation projects.

Conclusion and Future Work

DxMI demonstrates that viewing diffusion through the lens of IRL and dynamic programming can solve the efficiency-quality trade-off in generative modeling. While the framework currently requires at least 4 steps for high-quality results, future work may focus on reaching single-step generation and further automating hyperparameter tuning for temperature and energy regularization.

Summary updated

Abstract

We present a maximum entropy inverse reinforcement learning (IRL) approach for improving the sample quality of diffusion generative models, especially when the number of generation time steps is small. Similar to how IRL trains a policy based on the reward function learned from expert demonstrations, we train (or fine-tune) a diffusion model using the log probability density estimated from training data. Since we employ an energy-based model (EBM) to represent the log density, our approach boils down to the joint training of a diffusion model and an EBM. Our IRL formulation, named Diffusion by Maximum Entropy IRL (DxMI), is a minimax problem that reaches equilibrium when both models converge to the data distribution. The entropy maximization plays a key role in DxMI, facilitating the exploration of the diffusion model and ensuring the convergence of the EBM. We also propose Diffusion by Dynamic Programming (DxDP), a novel reinforcement learning algorithm for diffusion models, as a subroutine in DxMI. DxDP makes the diffusion model update in DxMI efficient by transforming the original problem into an optimal control formulation where value functions replace back-propagation in time. Our empirical studies show that diffusion models fine-tuned using DxMI can generate high-quality samples in as few as 4 and 10 steps. Additionally, DxMI enables the training of an EBM without MCMC, stabilizing EBM training dynamics and enhancing anomaly detection performance.
BibTeX
@inproceedings{yoon2024maximum,
  author = {Sangwoong Yoon and Himchan Hwang and Dohyun Kwon and Yung-Kyun Noh and Frank C. Park},
  title = {{Maximum Entropy Inverse Reinforcement Learning of Diffusion Models with Energy-Based Models}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2024},
  url = {https://openreview.net/forum?id=V0oJaLqY4E},
}
Figure