Regularizing Energy among Training Samples for Out-of-Distribution Generalization

Yiting Chen, Qitian Wu, Junchi YanICLR 2025 · OpenReview

Introduction

In machine learning, Energy-Based Models (EBMs) assign a scalar value—or "energy"—to data points. Traditionally, a low energy value indicates that the model "expects" the data, while high energy means the model is "surprised." Currently, energy is predominantly used for Out-of-Distribution (OOD) detection: if a sample exhibits high energy, it is likely an outlier.

However, this paper argues that the research community has largely ignored the energy differences within the training set itself. Standard training naturally assigns lower energy to frequent samples (the "head" of a distribution) and higher energy to rare ones (the "tail"). This energy imbalance causes the model to overfit to common patterns, leading to failures when encountering "subpopulation shifts" (e.g., a model trained mostly on white dogs failing to recognize a black dog) or "domain shifts" (e.g., transferring from photos to sketches).

By introducing Influence-Aware Energy Regularization (IAER), the authors provide a unified framework that explains why popular long-tail recognition tricks work and offers a novel approach to boosting model performance across subpopulation and domain shifts.

Methodology: Influence-Aware Energy Regularization

The paper makes two major contributions to the toolkit of energy regularization:

1. Theoretical Unification The authors prove that two common techniques for handling imbalanced data—data re-weighting (assigning more importance to rare samples) and margin control (forcing a larger gap between correct and incorrect class scores)—are fundamentally two sides of the same coin: implicit energy regularization.

2. Influence-Aware Energy Regularization (IAER) For complex scenarios where the boundary between "rare" and "common" samples is unknown (implicit shifts), the authors leverage Influence Functions. This technique calculates how a minute change in a training sample's energy affects the loss on a validation set.

Illustration of Influence-Aware Energy Regularization (IAER)
  • If increasing a sample's energy helps the model generalize better, IAER pushes that energy up.
  • If decreasing it helps, IAER pulls it down.
  • This dynamic allows the model to "self-correct" its energy distribution without requiring explicit labels for subpopulations.

Key Insights

  • The "Aha" Moment on Long-Tails: The paper demonstrates that state-of-the-art long-tail methods (like LDAM) produce a nearly flat energy distribution across classes. In contrast, standard Empirical Risk Minimization (ERM) training shows a massive energy bias toward frequent classes. This confirms that "leveling the energy playing field" is critical for handling data imbalance.
Average energy distribution of ERM vs. LDAM on CIFAR-LT datasets
  • Energy $\neq$ Prediction: Crucially, the authors show that you can alter a sample's energy value without changing the classifier's actual prediction for that sample. This makes energy regularization an "orthogonal" tool—it can be seamlessly added to almost any existing training method to improve robustness.
  • The Overfitting Connection: The researchers found that the energy of well-classified training points is often unstable during training. They conjecture that this "un-regularized" energy is a primary driver of model overfitting.

Empirical Results

The authors rigorously tested IAER across a massive range of benchmarks, including Long-Tail Recognition (CIFAR-10/100-LT, ImageNet-LT, iNaturalist 2018), Subpopulation Shift (SubpopBench), and Domain Generalization (DomainBed).

  • Boosting the Baseline: On CIFAR-10-LT (with a steep 100:1 imbalance ratio), adding IAER to a standard model boosted accuracy from 70.36% to 75.83%.
  • Worst-Case Performance: In subpopulation shifts, the primary goal is to improve the "worst-performing group." On the CMNIST benchmark, IAER improved the worst-group accuracy from a dismal 4.6% to 14.3%.
  • Domain Generalization: On the PACS dataset (generalizing across Photos, Art, Cartoons, and Sketches), IAER consistently improved upon standard ERM, reaching 86.6% accuracy.

Limitations and Future Work

While highly effective, IAER does come with constraints:

  • Computational Cost: Calculating Influence Functions is mathematically heavy. It requires approximating the Inverse Hessian Matrix, which is computationally expensive for very large models. Although the authors used stochastic estimation to accelerate this, it remains a bottleneck.
  • Validation Dependency: The method relies heavily on having a representative validation set to calculate "influence." If the validation set is skewed or poor, the energy regularization might be misguided.
  • Convexity Assumptions: The underlying theory of influence functions is most robust for convex loss functions. Extending this deeply into the highly non-convex landscape of deep neural networks remains an ongoing challenge for future research.

Summary updated

Abstract

The energy-based model provides a unified framework for various learning models where an energy value is assigned to each configuration of random variables based on probability. Recently, different methods have been proposed to derive an energy value out of the logits of a classifier for out-of-distribution (OOD) detection or OOD generalization. However, these methods mainly focus on the energy difference between in-distribution and OOD data samples, neglecting the energy difference among in-distribution data samples. In this paper, we show that the energy among in-distribution data also requires attention. We propose to investigate the energy difference between in-distribution data samples. Both empirically and theoretically, we show that previous methods for subpopulation shift (\emph{e.g.}, long-tail classification) such as data re-weighting and margin control apply implicit energy regularization and we provide a unified framework from the energy perspective. With the influence function, we further extend the energy regularization framework to OOD generalization scenarios where the distribution shift is more implicit compared to the long-tail recognition scenario. We conduct experiments on long-tail datasets, subpopulation shift benchmarks, and OOD generalization benchmarks to show the effectiveness of the proposed energy regularization.
BibTeX
@inproceedings{chen2025regularizing,
  author = {Yiting Chen and Qitian Wu and Junchi Yan},
  title = {{Regularizing Energy among Training Samples for Out-of-Distribution Generalization}},
  booktitle = {International Conference on Learning Representations},
  year = {2025},
  url = {https://openreview.net/forum?id=Lbx9zdURxe},
}
Figure