Abstract
Standard diffusion models often suffer from distribution shifts and blurry outputs when sampled in a small number of steps. This paper introduces DxMI (Diffusion by Maximum Entropy IRL), a framework that reinterprets diffusion models as reinforcement learning policies and energy-based models (EBMs) as reward functions. By employing a novel dynamic programming approach (DxDP) to bypass memory-intensive backpropagation through time, the authors achieve high-quality image generation in as few as 4 to 10 steps and set new benchmarks in anomaly detection.
Introduction: Beyond Behavioral Cloning
Standard diffusion models are typically trained using a method analogous to behavioral cloning: they learn to mimic a predefined "expert" trajectory that gradually transforms noise into data over 1,000+ steps. However, a significant "distribution shift" occurs when these models are accelerated for real-time generation (e.g., 10 steps). The model encounters intermediate states it never saw during its fine-grained training, leading to distorted samples.
While distillation and adversarial training (GANs) can mitigate this, they often require pre-trained teacher models or suffer from training instability. This work argues that Inverse Reinforcement Learning (IRL) provides a superior alternative: instead of merely mimicking steps, the model learns the underlying "reward" (data density) and optimizes its path to maximize that reward.
The DxMI Framework
The authors propose a minimax game between two components:
- Energy-Based Model (EBM): Acts as the reward function, estimating the log-probability density of the training data.
- Diffusion Model: Acts as the policy, attempting to generate samples that receive high rewards from the EBM while maximizing its own entropy to ensure diversity.
Efficient Optimization via DxDP
Updating a diffusion model via RL is traditionally computationally prohibitive due to the need to backpropagate gradients through every timestep. To solve this, the authors introduce DxDP (Diffusion by Dynamic Programming). By transforming the problem into an optimal control formulation, DxDP uses value functions (learned via the Bellman residual) to update the model locally at each timestep. This removes the need to backpropagate through the entire chain, significantly reducing GPU memory usage and stabilizing gradients.
Key Insights: Entropy and Stability
A critical finding of this research is that maximizing entropy does more than just ensure sample diversity; it stabilizes the training of the EBM itself. By incorporating entropy regularization, the EBM can be trained without the expensive Markov Chain Monte Carlo (MCMC) sampling that typically bottlenecks EBM research.
As shown in the synthetic 2D experiments, the temperature parameter $\tau$ (representing entropy) allows the model to capture the global structure of the data distribution more effectively than standard EBM training.
Experimental Results
The researchers evaluated DxMI across image generation and anomaly detection benchmarks:
- High-Quality Few-Step Generation: On CIFAR-10, DxMI achieved an FID of 3.19 in just 10 steps, significantly outperforming standard DDIM (13.36) and specialized policy gradient methods. Qualitative results on ImageNet 64×64 and LSUN Bedroom 256×256 showed superior structural integrity compared to Consistency Models.
- State-of-the-Art Anomaly Detection: On the MVTec-AD benchmark, DxMI achieved a mean Area Under the Curve (AUC) of 97.0% for detection and 97.1% for localization, surpassing previous state-of-the-art models like UniAD.
- Computational Efficiency: Training on CIFAR-10 requires less than 4 hours on four A100 GPUs, making the framework significantly more accessible than large-scale distillation projects.
Conclusion and Future Work
DxMI demonstrates that viewing diffusion through the lens of IRL and dynamic programming can solve the efficiency-quality trade-off in generative modeling. While the framework currently requires at least 4 steps for high-quality results, future work may focus on reaching single-step generation and further automating hyperparameter tuning for temperature and energy regularization.
Summary updated