Co-training Energy Functions and Neural Samplers

This direction is split into other directions: minimax energy optimization. Its last version stays readable and citable here.

Version 1 of 144 papers55 conceptsfrozen 2026-10-04

Energy-Based Models (EBMs) are mathematically elegant: they define a probability distribution by assigning a single scalar "energy" value to every possible input. Low energy means high probability; high energy means low probability. However, EBMs have historically been plagued by a fundamental "sampling bottleneck." To train them or generate data, you must navigate the energy landscape to find the low-energy valleys, typically using slow, iterative simulations like Markov Chain Monte Carlo (MCMC).

A growing body of research suggests a more efficient path: instead of relying on generic sampling algorithms, we should train a second neural network—a sampler or inference network—to learn how to navigate the energy landscape for us.

The Problem with MCMC

In traditional EBM training, we update the energy function so that real data points have low energy and "fantasy" samples (generated by the model) have high energy. Generating these fantasy samples usually requires Stochastic Gradient Langevin Dynamics (SGLD), where we start with noise and take many small steps down the energy gradient.

As Qing Yan, Yali Amit, and Zhisheng Xiao argue in EBMs Trained with Maximum Likelihood are Generator Models Trained with a Self-adverserial Loss, this process is essentially a "self-adversarial" game where the model's own gradient acts as a generator. However, MCMC is notoriously unstable, computationally expensive, and often gets stuck in local modes, failing to explore the full diversity of the data.

Amortizing Inference

The core insight of this research direction is amortization. Rather than running a fresh optimization for every sample, we train a parametric model (like a VAE, a Normalizing Flow, or a GAN-style generator) to map noise directly to low-energy states.

This creates a reciprocal learning loop:

  1. The Sampler provides the energy function with "negative samples" (contrastive examples).
  2. The Energy Function provides a training signal (a "reward" or "criticism") that tells the sampler how to improve.

Lifu Tu and Kevin Gimpel demonstrated this in Learning Approximate Inference Networks for Structured Prediction, where they replaced slow gradient descent at test time with a single forward pass of an inference network, achieving 60x speedups in structured tasks.

The Minimax Framework

Many researchers frame this joint training as a minimax game. In Learning to Draw Samples: With Application to Amortized MLE for Generative Adversarial Learning, Dilin Wang and Qiang Liu introduced SteinGAN, which uses Stein Variational Gradient Descent (SVGD) to train a generator to match an EBM's distribution.

This adversarial perspective allows EBMs to scale to high-dimensional data that was previously unreachable. For instance, Xuwang Yin and colleagues developed Dual Adversarial Training (DAT), which replaces unstable Langevin dynamics with a stable adversarial approach. By treating the sampler as a competitor that tries to find the "weak spots" in the energy landscape, they successfully scaled EBMs to high-resolution ImageNet data.

Bridging Different Generative Paradigms

This "co-training" strategy is not limited to GAN-like setups. It has been adapted to almost every major generative architecture:

Why It Works: Synergy and Stability

The reason this works better than training either model alone is a matter of synergy. The energy function is a powerful critic: it can capture complex, global dependencies (like the structural integrity of a molecule or the global coherence of an image) that are hard for a step-by-step generator to learn.

Conversely, the sampler is a powerful explorer: it provides the energy function with the exact "hard negatives" it needs to refine its boundaries. As noted in Balanced Training of Energy-Based Models with Adaptive Flow Sampling by Eric Moulines and Marylou Gabrié, using a companion model (like a Normalizing Flow) allows the system to make "global jumps," preventing the training process from getting trapped in local modes.

By turning the "sampling problem" into a "learning problem," this research direction transforms EBMs from theoretical curiosities into scalable, high-performance tools for structured prediction and generative modeling.

Papers in this version

History & citationVersion 1 · 2026-10-04

Split into other directions · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_co_training_energy_functions_and_neural__e5aff7b9,
  title = {Co-training Energy Functions and Neural Samplers},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/co-training-energy-functions-and-neural-samplers/rdr_e5aff7b9f03c47bbaac4907309162f39},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-21},
}

This exact version (version 1, frozen text):

@misc{lacuna_co_training_energy_functions_and_neural__e5aff7b9_v1,
  title = {Co-training Energy Functions and Neural Samplers},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/co-training-energy-functions-and-neural-samplers/rdr_e5aff7b9f03c47bbaac4907309162f39/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-21},
}