Unifying Classification and Generation via Joint Energy Models

39 concepts35 papers1 proposal

Standard deep learning splits tasks into two distinct paradigms: discriminative models (which learn the conditional distribution $p(y|x)$ to draw boundaries between classes) and generative models (which learn the data distribution $p(x)$ to synthesize new samples).

A standard classifier excels at drawing boundaries but is completely blind to whether an input belongs to the true data manifold. Consequently, when presented with out-of-distribution (OoD) data or adversarial perturbations, it confidently assigns incorrect labels. To fix this, we traditionally train separate architectures: a classifier for labels and a generative model (like a GAN or Diffusion model) for density estimation.

But what if a standard classifier could double as a generative model without changing its architecture? This is the core insight behind Joint Energy-Based Models (JEMs).

The Mathematical Bridge: Logits as Energy Scores

To extract a generative model from a standard classifier, we must reinterpret the denominator of the softmax function. Consider a standard neural network classifier $f(x)$ that outputs a vector of logits, where each component $f_y(x)$ corresponds to a class label $y$. The probability of a label given an input is parameterized via softmax:

$$p(y|x) = \frac{\exp(f_y(x))}{\sum_{y'} \exp(f_{y'}(x))}$$

An Energy-Based Model (EBM) defines the probability density of an input $x$ as $p(x) = \frac{\exp(-E(x))}{Z}$, where $E(x)$ is the energy function and $Z$ is the normalizing partition function. As noted by Senad Beadini et al. in Exploring the Connection between Robust and Generative Models and Fu Lin et al. in Likelihood Landscapes: A Unifying Principle Behind Many Adversarial Defenses, we can define the free energy of the input $x$ by looking directly at the denominator of the softmax function:

$$E(x) = -\log \sum_{y} \exp(f_y(x))$$

By doing this, the marginal distribution $p(x)$ is implicitly defined by the classifier's own internal representations. The joint distribution $p(x, y) = p(x)p(y|x)$ simplifies cleanly to:

$$p(x, y) = \frac{\exp(f_y(x))}{Z}$$

This simple mathematical reinterpretation allows a single neural network to simultaneously act as a discriminative classifier and a generative density estimator without altering its underlying architecture.

Smoothing the Energy Landscape

In a standard classifier, the network only learns to maximize the relative difference between logits. It does not care if the absolute values of the logits are high or low, meaning the energy landscape outside the training data is chaotic and unconstrained.

By training the network as a Joint Energy Model, we optimize both the conditional classification loss $\log p(y|x)$ and the generative log-likelihood $\log p(x)$. To maximize $\log p(x)$, the model must lower the energy (raise the logits) for real data points and raise the energy (lower the logits) for synthesized or noisy data points.

This dual optimization fundamentally alters the geometry of the model. As Fu Lin et al. demonstrate, optimizing for energy flattens the likelihood landscape around clean data. However, training JEMs can be notoriously unstable due to the need for Markov Chain Monte Carlo (MCMC) sampling to approximate the partition function gradient. To address this, Xiulong Yang et al. introduced SADA-JEM in Towards Bridging the Performance Gaps of Joint Energy-Based Models, utilizing Sharpness-Aware Minimization (SAM) to smooth the energy landscape and stabilize training, bridging the performance gap between hybrid models and specialized architectures.

Applications: Robustness, Calibration, and Privacy

Once a classifier understands the underlying data density, it gains several powerful capabilities:

Expanding the Paradigm: Clustering and Contrastive Learning

The principle of joint energy modeling extends far beyond standard image classification:

Alternative Architectures and Extensions

While JEMs typically reinterpret standard classifiers, related research explores alternative ways to embed energy functions into generative and predictive pipelines:

Information Theory and Representation Kernels

The mathematical elegance of joint energy modeling extends to structural representation and information theory:

Key Foundational Work

Your classifier is secretly an energy based model and you should treat it like one (2020) — contributed 2 concept elements to this direction.
Ramifications of Approximate Posterior Inference for Bayesian Deep Learning in Adversarial and Out-of-Distribution Settings (2020) — contributed 1 concept elements to this direction.
Likelihood Landscapes: A Unifying Principle Behind Many Adversarial Defenses (2020) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_joint_energy_based_modeling_5c052d50,
  title = {joint energy-based modeling},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/joint-energy-based-modeling/rdr_5c052d5030fb40dc896fd1da0794324f},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-19},
}

This exact version (version 1, frozen text):

@misc{lacuna_joint_energy_based_modeling_5c052d50_v1,
  title = {joint energy-based modeling},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/joint-energy-based-modeling/rdr_5c052d5030fb40dc896fd1da0794324f/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-19},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake