Abstract
Generative modeling requires a delicate balance between producing samples that are typical of the training data and maintaining sufficient diversity. Energy-Based Models (EBMs) often utilize Microcanonical Gradient Descent (MGDM) for sampling, but this process frequently leads to "entropy collapse"—a phenomenon where the distribution of generated samples shrinks excessively, resulting in repetitive outputs. This paper introduces Mean-Field Microcanonical Gradient Descent (MF-MGDM), a novel sampling technique that mitigates entropy collapse by applying energy constraints to the mean of a batch rather than to individual samples. Our theoretical and empirical results demonstrate that MF-MGDM preserves significantly more entropy and achieves lower reverse Kullback–Leibler (KL) divergence compared to standard MGDM across various domains, including synthetic time series and complex textures.
Introduction
The goal of generative modeling is to learn a distribution $p(x)$ that captures the characteristics of a dataset while allowing for the generation of novel, diverse samples. Energy-Based Models (EBMs) define this distribution via an energy function $\Phi(x)$, where low energy corresponds to high probability. A common sampling strategy is Microcanonical Gradient Descent (MGDM), which initializes samples as random noise and uses gradient descent to move them toward a manifold of a specific target energy $\alpha$.
However, MGDM is inherently contractive. As the gradient descent progresses, the volume of the sample distribution tends to shrink, leading to a loss of diversity known as entropy collapse. In this state, the model produces samples that satisfy the energy constraint but are nearly identical, failing to represent the full breadth of the target distribution. This paper addresses this limitation by proposing a "mean-field" regularization approach.
Methods: From Individual to Batch Constraints
The core innovation of this work is the shift from individual sample constraints to aggregate constraints. In standard MGDM, each sample $x^{(n)}$ in a batch is updated independently to minimize its distance to the target energy $\alpha$: $$ \mathcal{L}_{indiv} = \sum_{n=1}^N (\Phi(x^{(n)}) - \alpha)^2 $$ This aggressive individual enforcement is what drives the distribution to collapse.
In contrast, Mean-Field MGDM (MF-MGDM) updates a batch of $N$ samples simultaneously by targeting the mean energy of the entire batch: $$ \mathcal{L}_{batch} = (\bar{\Phi}(\mathbf{x}) - \alpha)^2, \text{ where } \bar{\Phi}(\mathbf{x}) = \frac{1}{N} \sum_{n=1}^N \Phi(x^{(n)}) $$ This "weak coupling" allows individual samples to deviate from the target energy as long as the group average remains correct. This relaxation provides the necessary "room to breathe" for individual particles, preserving the volume of the initial distribution.
The authors further provide a theoretical foundation for this method, deriving a lower bound on the entropy of the sample distribution. They show that the rate of entropy loss is inversely proportional to the batch size $N$, suggesting that larger batches provide better regularization.
Results and Discussion
The effectiveness of MF-MGDM was evaluated on several datasets, including synthetic autoregressive processes (AR), Cox-Ingersoll-Ross (CIR) models, and 2D texture synthesis.
Quantitative Performance
In synthetic experiments where the true density is known, MF-MGDM consistently outperformed standard MGDM in terms of reverse KL divergence. For instance, in an AR(0.1) process, the minimum KL divergence achieved by MF-MGDM was 0.09, a massive improvement over the 2.76 achieved by standard MGDM.
Entropy Stability
A key advantage of MF-MGDM is its stability over long gradient descent trajectories. While standard MGDM requires careful early stopping to avoid total entropy collapse, MF-MGDM allows the descent to run to convergence. As shown in Figure 5, the entropy in MF-MGDM levels off at a much higher value, and the KL divergence continues to decrease or remains stable, whereas in MGDM, the KL divergence eventually spikes as entropy plummets.
Visual Diversity in Textures
On the "bubbles" texture dataset, MF-MGDM produced samples that were visually indistinguishable from the training data in terms of local features but exhibited significantly higher global variety across the batch. Standard MGDM, by contrast, tended to produce very similar-looking "bubble" patterns across all samples in the batch after many iterations.
Conclusion
Mean-Field Microcanonical Gradient Descent offers a simple yet powerful modification to energy-based sampling. By relaxing individual constraints in favor of batch-level averages, it effectively combats entropy collapse without sacrificing sample quality.
Future work includes extending the method to non-differentiable constraints, such as those based on order statistics, and improving the stability of the descent inversion for applications in normalizing flows. Additionally, exploring the optimal tuning of the microcanonical set width $\epsilon$ remains an important open question for achieving the best possible fit to complex data distributions.
Summary updated