Regularizing Empirical Risk with f-Divergences

This direction is merged into another direction: partition function intractability (partition function estimation). Its last version stays readable and citable here.

Version 1 of 15 papers22 conceptsfrozen 2026-10-04

In standard Empirical Risk Minimization (ERM), we seek a model that minimizes the average loss over a training set. However, without constraints, ERM often overfits or produces solutions that are physically or logically inconsistent with our prior knowledge. To steer the optimization, researchers often add a regularization term that penalizes the distance between the learned model and a reference distribution (a "prior").

Recent work by Francisco Daunas, H. Vincent Poor, Iñaki Esnaola, and Samir M. Perlaza in Equivalence of Empirical Risk Minimization to Regularization on the Family of $f- \text{Divergences}$ provides a unified mathematical framework for understanding this process. They demonstrate that using the broad family of $f$-divergences—which includes the Kullback-Leibler (KL) divergence, Hellinger distance, and Total Variation—creates a fundamental "support constraint" on the learning process.

The Absolute Inductive Bias

The most striking property of $f$-divergence regularization is the support constraint. In probability theory, the "support" of a distribution is the set of outcomes that have a non-zero probability.

Daunas et al. prove that when you regularize ERM with an $f$-divergence relative to a reference measure, the optimal solution is mathematically forbidden from assigning any probability to outcomes that the reference measure deems impossible.

Think of the reference measure as a map of "allowable reality." If your prior knowledge (the reference) says a certain state is impossible, no amount of training data can force the model to predict that state. This makes the inductive bias of the prior absolute. Unlike other forms of regularization that merely "nudge" the model, $f$-divergence regularization acts as a hard boundary that training data cannot override.

The Equivalence Theorem

Beyond the support constraint, the researchers uncover a deep symmetry between the loss function and the choice of regularization. They establish an equivalence theorem: changing the type of $f$-divergence used for regularization is mathematically identical to transforming the underlying loss function.

This suggests that the "shape" of our uncertainty (the divergence) and the "cost" of our errors (the loss) are two sides of the same coin. Specifically:

  1. Interchangeability: If you find that a specific loss function is not performing well, you might not need to change the loss itself. Instead, you could achieve the same effect by changing the $f$-divergence used to regularize the model.
  2. Unified Optimization: This equivalence allows researchers to solve complex regularized problems by mapping them back to a simpler, transformed ERM problem.

Why It Works: Intuition from Geometry

To understand why this works, imagine the space of all possible models. ERM tries to find the point in this space that minimizes loss. Regularization acts like a rubber band, pulling the solution toward a reference point.

The $f$-divergence family determines the "stiffness" and "shape" of that rubber band. Because $f$-divergences are defined by convex functions, they reshape the optimization landscape. Daunas et al. show that this reshaping is so consistent that the effect of the "rubber band" can be perfectly absorbed into the "loss landscape" itself.

By proving this equivalence, the authors provide a toolkit for ensuring that models remain consistent with structural constraints—such as physical laws or logical bounds—while offering a simplified way to tune the trade-off between data-driven learning and prior-driven constraints.

Papers in this version

History & citationVersion 1 · 2026-10-04

Merged into another direction · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_regularizing_empirical_risk_with_f_diver_70aa4fc8,
  title = {Regularizing Empirical Risk with f-Divergences},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/regularizing-empirical-risk-with-f-divergences/rdr_70aa4fc8e8b24eb6ac5f7feab963cfc0},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-21},
}

This exact version (version 1, frozen text):

@misc{lacuna_regularizing_empirical_risk_with_f_diver_70aa4fc8_v1,
  title = {Regularizing Empirical Risk with f-Divergences},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/regularizing-empirical-risk-with-f-divergences/rdr_70aa4fc8e8b24eb6ac5f7feab963cfc0/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-21},
}