Standard deep learning splits tasks into two distinct paradigms: discriminative models (which learn the conditional distribution $p(y|x)$ to draw boundaries between classes) and generative models (which learn the data distribution $p(x)$ to synthesize new samples).
A standard classifier excels at drawing boundaries but is completely blind to whether an input belongs to the true data manifold. Consequently, when presented with out-of-distribution (OoD) data or adversarial perturbations, it confidently assigns incorrect labels. To fix this, we traditionally train separate architectures: a classifier for labels and a generative model (like a GAN or Diffusion model) for density estimation.
But what if a standard classifier could double as a generative model without changing its architecture? This is the core insight behind Joint Energy-Based Models (JEMs).
The Mathematical Bridge: Logits as Energy Scores
To extract a generative model from a standard classifier, we must reinterpret the denominator of the softmax function. Consider a standard neural network classifier $f(x)$ that outputs a vector of logits, where each component $f_y(x)$ corresponds to a class label $y$. The probability of a label given an input is parameterized via softmax:
$$p(y|x) = \frac{\exp(f_y(x))}{\sum_{y'} \exp(f_{y'}(x))}$$
An Energy-Based Model (EBM) defines the probability density of an input $x$ as $p(x) = \frac{\exp(-E(x))}{Z}$, where $E(x)$ is the energy function and $Z$ is the normalizing partition function. As noted by Senad Beadini et al. in Exploring the Connection between Robust and Generative Models and Fu Lin et al. in Likelihood Landscapes: A Unifying Principle Behind Many Adversarial Defenses, we can define the free energy of the input $x$ by looking directly at the denominator of the softmax function:
$$E(x) = -\log \sum_{y} \exp(f_y(x))$$
By doing this, the marginal distribution $p(x)$ is implicitly defined by the classifier's own internal representations. The joint distribution $p(x, y) = p(x)p(y|x)$ simplifies cleanly to:
$$p(x, y) = \frac{\exp(f_y(x))}{Z}$$
This simple mathematical reinterpretation allows a single neural network to simultaneously act as a discriminative classifier and a generative density estimator without altering its underlying architecture.
Smoothing the Energy Landscape
In a standard classifier, the network only learns to maximize the relative difference between logits. It does not care if the absolute values of the logits are high or low, meaning the energy landscape outside the training data is chaotic and unconstrained.
By training the network as a Joint Energy Model, we optimize both the conditional classification loss $\log p(y|x)$ and the generative log-likelihood $\log p(x)$. To maximize $\log p(x)$, the model must lower the energy (raise the logits) for real data points and raise the energy (lower the logits) for synthesized or noisy data points.
This dual optimization fundamentally alters the geometry of the model. As Fu Lin et al. demonstrate, optimizing for energy flattens the likelihood landscape around clean data. However, training JEMs can be notoriously unstable due to the need for Markov Chain Monte Carlo (MCMC) sampling to approximate the partition function gradient. To address this, Xiulong Yang et al. introduced SADA-JEM in Towards Bridging the Performance Gaps of Joint Energy-Based Models, utilizing Sharpness-Aware Minimization (SAM) to smooth the energy landscape and stabilize training, bridging the performance gap between hybrid models and specialized architectures.
Applications: Robustness, Calibration, and Privacy
Once a classifier understands the underlying data density, it gains several powerful capabilities:
- Out-of-Distribution Detection and Calibration: Standard classifiers suffer from overconfidence on unseen data. By assigning absolute energy scores rather than relative probabilities, JEMs can easily flag anomalies. John Mitros et al. show in Ramifications of Approximate Posterior Inference for Bayesian Deep Learning in Adversarial and Out-of-Distribution Settings that leveraging the underlying density improves calibration, while Yaqian Hao et al. leverage this in Exploring Energy-Based Models for Out-of-Distribution Detection in Dialect Identification to reliably identify unknown speech patterns.
- Adversarial Robustness: Adversarial attacks exploit the sharp, unconstrained boundaries of discriminative models. Yao Zhu et al. explore this in Towards Understanding the Generative Capability of Adversarially Robust Classifiers, showing that robust training inherently shapes the energy landscape to align with the true data manifold, allowing classifiers to generate realistic images via gradient-based sampling.
- Privacy Protection: Classifiers often leak information about their training sets, making them vulnerable to Membership Inference Attacks. In Protect privacy of deep classification networks by exploiting their generative power, Jiyu Chen et al. exploit the generative power of JEMs to synthesize privacy-preserving datasets that mimic the original distribution without exposing individual training samples.
- Guided Generation: Paul Kuo-Ming Huang et al. demonstrate in Score-based Conditional Generation with Fewer Labeled Data by Self-calibrating Classifier Guidance that treating a classifier as an EBM allows its logits to provide unconditional density estimates, which can guide score-based diffusion models when labeled data is scarce.
Expanding the Paradigm: Clustering and Contrastive Learning
The principle of joint energy modeling extends far beyond standard image classification:
- Self-Supervised Clustering: Emanuele Sansone et al. show in Unifying Self-Supervised Clustering and Energy-Based Models that treating the softmax denominator as an energy score aligns representation learning with density estimation, creating a single network that clusters data while generating new samples.
- Contrastive Learning: In Energy-Based Contrastive Learning of Visual Representations, Beomsu Kim et al. show that standard contrastive losses can be generalized into an energy framework that models the joint density of positive pairs, drastically accelerating training and reducing dependency on massive batch sizes.
- Domain Adaptation: Xiaofeng Liu et al. apply this to unsupervised domain adaptation in Energy-constrained Self-training for Unsupervised Domain Adaptation, minimizing the free energy of unlabeled target samples to regularize noisy pseudo-labels.
- Multi-Task Physics Surrogates: The versatility of this framework is highlighted by Taoli Cheng et al. in Versatile Energy-Based Probabilistic Models for High Energy Physics, where a single energy-based network simultaneously handles particle jet generation, classification, and anomaly detection without introducing spurious correlations.
Alternative Architectures and Extensions
While JEMs typically reinterpret standard classifiers, related research explores alternative ways to embed energy functions into generative and predictive pipelines:
- Latent Space Energy Models: Rather than sampling in high-dimensional pixel space, Omead Pooladzandi et al. leverage a latent energy-based prior in Generating High Fidelity Synthetic Data via Coreset selection and Entropic Regularization to filter out low-quality synthetic samples, ensuring they boost semi-supervised learning performance. Similarly, Mitch Hill et al. propose the Hat EBM in Learning Probabilistic Models from Generator Latent Spaces with Hat EBM, which layers an energy function on top of deterministic generators to convert them into full probabilistic models.
- Out-of-Equilibrium Generators: To bypass slow MCMC sampling, Alessandra Carbone et al. develop a framework in Fast and Functional structured data generator that alternates between clamping data and labels to train out-of-equilibrium models that classify and generate high-quality samples in very few MCMC steps.
- Concept Bottlenecks: In the realm of interpretability, Xinyue Xu et al. introduce Energy-Based Concept Bottleneck Models: Unifying Prediction, Concept Intervention, and Probabilistic Interpretations, where energy functions capture complex correlations between inputs, human-understandable concepts, and labels, turning interventions into simple conditional probability queries.
Information Theory and Representation Kernels
The mathematical elegance of joint energy modeling extends to structural representation and information theory:
- Neural Fisher Kernels: Ruixiang ZHANG et al. introduce the Neural Fisher Kernel in Learning Representation from Neural Fisher Kernel with Low-rank Approximation, unifying generative models and classifiers under an energy-based framework to extract compact, low-rank data embeddings.
- Rate-Distortion Estimation: In information theory, Shitong Wu et al. utilize a single neural energy function in Estimating Rate-Distortion Functions Using the Energy-Based Model to simultaneously model the marginal distribution of compressed data and the conditional mapping of the source, solving high-dimensional compression limits.
- Stochastic Simulations: Paula Cordero Encinar et al. demonstrate in Deep Optimal Sensor Placement for Black Box Stochastic Simulations that joint energy models can capture the joint probability between system inputs and outputs, enabling both forward simulation and inverse inference for optimal sensor placement.
Key Foundational Work
Your classifier is secretly an energy based model and you should treat it like one (2020) — contributed 2 concept elements to this direction.
Ramifications of Approximate Posterior Inference for Bayesian Deep Learning in Adversarial and Out-of-Distribution Settings (2020) — contributed 1 concept elements to this direction.
Likelihood Landscapes: A Unifying Principle Behind Many Adversarial Defenses (2020) — contributed 1 concept elements to this direction.