Introduction
Energy-Based Models (EBMs) offer a flexible framework for representing probability distributions by assigning a scalar "energy" value to inputs. In this paradigm, low energy corresponds to high probability (e.g., natural images), while high energy corresponds to low probability (e.g., noise). Despite their theoretical appeal, EBMs are notoriously difficult to train. The standard training algorithm, Contrastive Divergence (CD), often suffers from instability, requiring restrictive techniques like gradient clipping or spectral normalization to prevent divergence. This paper identifies a critical "missing" gradient term in the classic CD algorithm and proposes a stabilized training recipe that enables high-resolution image generation and the use of modern architectural components like self-attention.
Methodology: Stabilizing the Energy Landscape
The core contribution of this work is the identification and inclusion of a previously ignored gradient term. In standard CD, the gradient of the log-likelihood is approximated by the difference between the energy of real data and the energy of model-generated "hallucinations" produced via Markov Chain Monte Carlo (MCMC) sampling. However, this approximation ignores the fact that changing model parameters also shifts the distribution of the MCMC samples themselves.
The KL Loss and Entropy Estimation
The authors re-introduce this missing term as a KL Divergence Loss ($L_{KL}$). This term decomposes into two functional parts: one that minimizes the energy of generated samples and another that maximizes their entropy. To prevent the "mode collapse" common in generative models—where samples cluster around a single point—the authors employ a non-parametric Nearest-Neighbor Entropy Estimator. This ensures that the generated samples remain diverse and cover the data distribution effectively.
Multi-scale Architecture and Data Augmentation
To capture both global structure and fine-grained details, the model uses a multi-scale architecture where the total energy is the sum of energies calculated at multiple resolutions (e.g., full, 1/2, and 1/4 size). Furthermore, the authors treat data augmentations (like horizontal flips) as valid MCMC transitions. This "Data Augmentation Transition" helps the Langevin sampler jump between different modes of the energy landscape, preventing it from getting stuck in local minima.
Experimental Results
The proposed framework was evaluated across several benchmarks, including CIFAR-10, CelebA-HQ (128x128), and LSUN Bedroom.
Generative Performance
The stabilized training allowed for the inclusion of self-attention and layer normalization, leading to significant improvements in sample quality. On CIFAR-10, the model achieved an FID of 25.1. More impressively, it successfully scaled to 128x128 resolutions on CelebA-HQ and LSUN Bedroom, reaching an FID of 28.78 on the former.
Out-of-Distribution Detection
EBMs are naturally suited for Out-of-Distribution (OOD) detection because they explicitly model the energy (unnormalized log-probability) of the input. The improved training recipe further enhanced this capability. When trained on CIFAR-10, the model outperformed likelihood-based models like PixelCNN++ and Glow in identifying samples from SVHN or texture datasets.
Conclusion and Future Work
By incorporating the "missing" KL gradient term and leveraging multi-scale processing with augmented MCMC transitions, this work significantly advances the stability and scalability of Energy-Based Models. While MCMC-based sampling remains slower than GAN or VAE inference, the improved robustness opens the door for EBMs to be used with more complex architectures and in diverse domains beyond image synthesis, such as reinforcement learning and structural biology. Future research will focus on further accelerating the sampling process to enable megapixel-scale generation.
Summary updated