Exponential-Family Harmoniums with Neural Sufficient Statistics

Azwar Abdulsalam, Joseph G. MakinAAAI 2025 · DOI

This paper introduces Neural-Network Exponential-Family Harmoniums (NN-EFHs), a generative model that bridges the gap between classical undirected graphical models and modern deep learning. By allowing deep neural networks to define the "features" (sufficient statistics) of the model, the authors achieve exact, efficient inference while producing high-quality image samples that significantly outperform standard energy-based models and previous restricted Boltzmann machine variants.

The Challenge: Generation vs. Inference

Generative modeling usually involves a difficult trade-off between generation (creating new data) and inference (understanding the latent variables behind existing data).

  • Directed models (like VAEs or Diffusion models) are excellent at generation but often require "recognition models" to approximate inference. These approximations can be inaccurate or computationally expensive.
  • Undirected models (like Restricted Boltzmann Machines, or RBMs) offer exact and computationally "cheap" inference. However, they have historically struggled with complex data like high-resolution images because they are mathematically restricted to simple "exponential family" distributions (like Bernoulli or Gaussian). These simple distributions fail to capture the intricate, translation-invariant patterns—such as edges and textures—found in natural images.

The authors set out to modernize the Exponential-Family Harmonium (EFH)—the generalized version of the RBM—so it can handle modern computer vision tasks without losing its core advantage: exact inference.

Methods: Neural Sufficient Statistics

The core innovation is the NN-EFH. Instead of using fixed, simple sufficient statistics, the authors parameterize them using deep convolutional neural networks (CNNs).

The NN-EFH architecture uses a deep neural network to compute sufficient statistics $U(\hat{x}, \phi)$, enabling an exact posterior $\hat{p}(\hat{z}|\hat{x}; \theta)$ while maintaining a bilinear energy structure.
  1. Neural Sufficient Statistics: The model's energy function is defined such that the features of the observed data are computed by a CNN. This allows the model to learn complex visual features while keeping the mathematical relationship between latent and visible units bilinear.
  2. Exact Posterior: Because of this bilinear structure, the "posterior" distribution (the probability of latent variables given an image) remains a simple product of Bernoullis. This means inference is still exact and very fast.
  3. Gibbs-Langevin Sampling: Since the "emission" distribution (the probability of an image given latent variables) is no longer a standard distribution, the authors use a hybrid sampling scheme. They combine block Gibbs sampling (to jump between latent and visible spaces) with Langevin dynamics (using gradients to refine the image).
  4. Energy Shaping: They include a quadratic term in the energy function to encourage convexity far from the data, which helps the sampling process converge faster.

Key Insights: The "Jump" Advantage

A critical finding of the research is that Gibbs-Langevin sampling is much more efficient than standard Langevin dynamics alone.

Comparison of sampling dynamics: (a) standard Langevin dynamics crawling through data space versus (b) the Gibbs-Langevin "jump" advantage, where sampling latent variables allows for larger steps across the energy landscape.

While Langevin dynamics crawls through the data space using gradients, the Gibbs step allows the model to "jump" across the energy landscape by sampling latent variables. The authors observed that the distance traveled in the data space was 10x greater with this hybrid approach, leading to faster convergence and better exploration of the data manifold.

Other key insights include:

  • Breaking Symmetry: In traditional EFHs, visible and latent units are symmetric. By using a deep network for the visible units' statistics, the authors break this symmetry to favor the complex structure of images while keeping the latent side simple for easy inference.
  • Superior Representations: The latent variables learned by NN-EFHs are highly meaningful. When used for downstream tasks, features from an NN-EFH were nearly twice as accurate for classification as those from a standard Energy-Based Model (EBM).

Key Results

The NN-EFH represents the first time an RBM/EFH-style architecture has successfully generated realistic, diverse samples from challenging datasets like CIFAR-10.

  • FID Scores: On CelebA-HQ, the NN-EFH achieved an FID of 11.3, significantly better than the EBM baseline (22.7), VAEs (38.76), and even the flow-based model Glow (23.32).
  • CIFAR-10 Performance: The model achieved an FID of 32.1, which is competitive with noise-conditional score networks (NCSN) despite using simpler neural architectures and fewer sampling steps.
  • Classification Accuracy: On CIFAR-10, a simple linear classifier trained on NN-EFH features reached 53% accuracy, compared to only 29% for a standard EBM.

Limitations and Future Work

While promising, the model still faces challenges. Generation requires multiple steps of Langevin dynamics and Gibbs sampling, making it slower than "one-pass" models like GANs. Additionally, scaling to megapixel-scale images remains a hurdle for most energy-based models. Future work may involve incorporating contrastive losses to further improve the quality of the learned latent representations.

Summary updated

BibTeX
@inproceedings{abdulsalam2025exponential,
  author = {Azwar Abdulsalam and Joseph G. Makin},
  title = {{Exponential-Family Harmoniums with Neural Sufficient Statistics}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2025},
  doi = {10.1609/aaai.v39i15.33677},
}
Figure