Score-based generative models and Energy-Based Models (EBMs) have become central to modern machine learning because they bypass the "partition function problem." By focusing on the score function—the gradient of the log-density $\nabla_x \log p(x)$—researchers can train models without ever calculating the intractable normalizing constant that plagues traditional maximum likelihood estimation.
However, this convenience comes with a fundamental "blind spot." As Li K. Wenliang demonstrates in Blindness of score-based methods to isolated components and mixing proportions, score-based methods are inherently unable to perceive the relative weights (mixing proportions) of distant modes in a distribution.
The Locality of the Score Function
To understand why this happens, consider a distribution $p(x)$ composed of two isolated clusters (modes) separated by a vast region of near-zero probability. The score function $\nabla_x \log p(x)$ acts like a signpost, pointing toward the nearest peak of high density.
If you are standing on the slope of Mode A, the score tells you how to climb toward its center. Crucially, the local slope of Mode A is entirely independent of how much total probability mass is assigned to Mode B. Whether Mode B contains 1% or 99% of the data, the "shape" of Mode A—and thus its score—remains the same.
As Yang Song and Diederik P Kingma explain in How to Train Your Energy-Based Models, because the score function only provides local information, the model has no global mechanism to reconcile the mass of disconnected components. This leads to what Tobias Schröder and Yingzhen Li call "nearsightedness": the model captures local structures perfectly but fails to learn the global distribution weights.
Consequences Across the Pipeline
This blindness isn't just a theoretical curiosity; it creates practical failures in training, sampling, and testing:
- Inaccurate Generative Modeling: Alexia Jolicoeur-Martineau and colleagues note in Adversarial score matching and improved sampling for image generation that while score-based models are excellent at identifying all modes (avoiding the total mode collapse seen in GANs), they struggle to match the exact probability mass of those modes.
- Failed Statistical Tests: Goodness-of-fit tests like the Kernelized Stein Discrepancy (KSD) rely on scores. Xing Liu and Andrew Duncan show in Using Perturbation to Improve Goodness-of-Fit Tests based on Kernelized Stein Discrepancy that these tests can be "fooled" by models with correct mode locations but wildly incorrect mixing weights.
- Scientific Inaccuracy: In chemistry, Adam Foster and Jonas Köhler found that score matching captures local electron density features but fails to partition charge accurately between spatially separated atoms in a molecule (Highly Accurate Real-space Electron Densities with Neural Networks).
Healing the Blindness
The research community has proposed several strategies to reintroduce global awareness into score-based frameworks:
1. Bridging the Gaps
If the problem is that the score is uninformative in low-density regions between modes, one solution is to "fill" those gaps. Mingtian Zhang and Oscar Key propose Mixture Fisher Divergence (MFD) in Towards Healing the Blindness of Score Matching. By mixing the target and model distributions with a shared auxiliary "bridge" distribution that spans the space between modes, the objective can finally capture global mass allocation.
2. Energy-Based Discrepancies
Instead of matching gradients (scores), some researchers suggest matching energy levels directly. Tobias Schröder et al. introduced Energy Discrepancy (ED) in Energy Discrepancies: A Score-Independent Loss for Energy-Based Models. By comparing the energy of data points to perturbed samples, the model learns the relative heights of the energy landscape, which implicitly encodes the mixing weights.
3. Data-Based Initialization
A surprisingly simple fix for sampling is to start the process from the data itself. Frederic Koehler and Thuy-Duong Vuong prove in Sampling Multimodal Distributions with the Vanilla Score: Benefits of Data-Based Initialization that if you initialize Langevin diffusion with real data points and stop early, you can accurately sample from multimodal distributions even if the learned score function is "flawed" in the gaps between modes.
4. Density Estimation via Time-Reversal
Newer frameworks like the Radon-Nikodym Estimator (RNE), developed by Jiajun He and Yuanqi Du, attempt to reveal the hidden densities within diffusion models. By leveraging the relationship between a forward process and its time-reversal, RNE provides a way to estimate absolute densities rather than just local gradients, enabling more accurate training of EBMs (RNE: a plug-and-play framework for diffusion density estimation and inference-time control).