Contrastive Divergence for Energy Based Models

You are reading version 1 of this direction, frozen 2026-10-04. The latest version is version 2.

Version 1 of 240 papers42 conceptsfrozen 2026-10-04

In the early 2000s, a significant challenge in machine learning was training undirected graphical models—often called Energy-Based Models (EBMs). Unlike directed models (like Bayesian networks), undirected models define the probability of a state through an energy function $E(x)$. The lower the energy, the higher the probability.

The mathematical catch is the "partition function" $Z$: a normalization constant that ensures all probabilities sum to one. To calculate $Z$, you must sum the energy of every possible state in the system. For high-dimensional data like images or text, this sum is computationally impossible. Geoffrey E. Hinton and his colleagues proposed a shortcut that bypassed this "normalization nightmare," leading to the development of Contrastive Divergence (CD).

The Intuition: Stop the Drift

To train an EBM, we want to lower the energy of real data points (the "positive phase") and raise the energy of points the model thinks are likely but are actually fake (the "negative phase").

Traditionally, finding these "fake" points required running a Markov Chain Monte Carlo (MCMC) sampler until it reached thermal equilibrium—a state where the samples perfectly represent the model's current beliefs. This is incredibly slow.

The core insight of Contrastive Divergence, introduced in Discovering Multiple Constraints that are Frequently Approximately Satisfied, is that we don't need to wait for equilibrium. Instead, we start a Markov chain at a real data point and run it for just a few steps (often just one). If the model is accurate, the "reconstructed" data point shouldn't drift far from the original. If it does drift, we adjust the model's weights to pull the energy of the original data down and push the energy of the drifted reconstruction up.

Products of Experts and Harmoniums

This "short-run" MCMC approach allowed researchers to train much more complex architectures. Geoffrey E. Hinton used this to develop Products of Experts (PoE), where multiple simple models (experts) each constrain different dimensions of the data. In Recognizing Hand-written Digits Using Hierarchical Products of Experts, Guy Mayraz and Hinton showed that these models could learn the complex distributions of handwritten digits by stacking Restricted Boltzmann Machines (RBMs).

The framework was further generalized by Max Welling and others into Exponential Family Harmoniums (EFHs). As detailed in Exponential Family Harmoniums with an Application to Information Retrieval, EFHs extend the RBM structure to any distribution in the exponential family (like Gaussian or Poisson), enabling the modeling of diverse data types like word counts in documents. Similarly, the Product of Student-t (PoT) model, explored in Learning Sparse Topographic Representations with Products of Student-t Distributions, used CD to learn sparse, brain-like visual features from natural images.

Persistent Chains and Stability

While standard CD starts the Markov chain from the data at every training step, it has a flaw: it doesn't provide a perfect estimate of the gradient. Tijmen Tieleman proved in On the Convergence Properties of Contrastive Divergence that CD is not technically the gradient of any single objective function, which can occasionally lead to unstable learning.

To fix this, researchers developed Persistent Contrastive Divergence (PCD). Instead of restarting the chain from the data at each step, PCD maintains "fantasy particles" that persist across iterations. As the model parameters change slowly, these particles stay close to the model's evolving equilibrium. This technique, discussed in Ratio Divergence Learning Using Target Energy in Restricted Boltzmann Machines, provides a more accurate gradient estimate and is now a standard tool for training RBMs and modern energy-based packages like adabmDCA 2.0.

Beyond Generative Modeling

The efficiency of CD-based training opened doors beyond simple density estimation:

By replacing the impossible task of calculating global normalization with the local task of "checking for drift," Contrastive Divergence transformed undirected graphical models from theoretical curiosities into scalable, practical tools for unsupervised learning.

Papers in this version

History & citationVersion 2 · 2026-10-04

Current · Permanent link

  1. Version 2 (latest) · 2026-10-04 · 27 papers
  2. Version 1 · 2026-10-04 · 40 papers
Cite this direction

Latest version (a living page):

@misc{lacuna_contrastive_divergence_fe7b0593,
  title = {contrastive divergence},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/contrastive-divergence/rdr_fe7b059370c84121a0ad98ad13633694},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-21},
}

This exact version (version 1, frozen text):

@misc{lacuna_contrastive_divergence_for_energy_based__fe7b0593_v1,
  title = {Contrastive Divergence for Energy Based Models},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/contrastive-divergence-for-energy-based-models/rdr_fe7b059370c84121a0ad98ad13633694/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-21},
}