Entropic Regularization Bias in Optimal Transport

33 concepts27 papers1 proposal

Optimal Transport (OT) is a powerful geometric framework for comparing probability distributions. By calculating the minimal cost of moving mass from distribution $\alpha$ to distribution $\beta$, it respects the underlying geometry of the underlying space. However, solving the classic unregularized Monge-Kantorovich problem is computationally demanding, traditionally requiring $O(n^3 \log n)$ time.

To make OT practical for machine learning, researchers frequently add an entropic regularization term to the objective function. This modification transforms a rigid linear program into a strictly convex optimization problem that can be solved in near-linear time using the elegant, GPU-friendly Sinkhorn algorithm.

Yet, this computational speedup comes with a hidden cost: entropic regularization bias. While entropic OT makes large-scale data alignment possible, it systematically alters the geometry of the problem, introducing a pervasive "blurring" effect that degrades the precision of the resulting transport plans, maps, and averages.

Why Entropic Regularization Blurs

To understand why entropic regularization introduces bias, we must look at how it changes the optimization landscape. The standard optimal transport problem seeks a transport plan matrix $\pi$ that minimizes the expected cost:

$$\min_{\pi \in \Pi(\alpha, \beta)} \sum_{i,j} \pi_{ij} C_{ij}$$

Because this is a linear program over a bounded polytope, the optimal solution naturally tends to land on the vertices of the polytope. This means the true transport plan is highly sparse; it favors deterministic, point-to-point assignments (often called Monge maps).

When we introduce entropic regularization with a smoothing parameter $\varepsilon > 0$, the objective becomes:

$$\min_{\pi \in \Pi(\alpha, \beta)} \sum_{i,j} \pi_{ij} C_{ij} - \varepsilon H(\pi)$$

where $H(\pi) = -\sum_{i,j} \pi_{ij} \log \pi_{ij}$ is the Shannon entropy of the transport plan.

The negative entropy term $-H(\pi)$ acts as a penalty on certainty. Maximizing entropy means forcing the elements of $\pi$ to be as uniform and spread out as possible. Consequently, the regularizer pulls the optimal solution away from the sharp vertices of the transport polytope and deep into its interior.

Instead of mapping a discrete point in $\alpha$ to a single unique point in $\beta$, the regularized plan diffuses the mass across a wide neighborhood. As noted by Laetitia Chapel et al. in Unbalanced Optimal Transport through Non-negative Penalized Linear Regression, this loss of sparsity destroys the interpretability of discrete correspondences, replacing crisp point-to-point matching with a hazy, probabilistic smear.

The Three Symptoms of Entropic Bias

This mathematical diffusion manifests as a systematic bias across three major applications of optimal transport.

1. The Self-Distance Anomaly

In pure geometry, the distance between any object and itself must be zero. However, entropic regularization violates this axiom. Because the entropy term penalizes sparse identity mappings, the regularized cost of transporting a distribution to itself is strictly positive ($OT_\varepsilon(\alpha, \alpha) > 0$).

As Hui LIN et al. demonstrate in Direct Measure Matching for Crowd Counting, this mathematical artifact creates an "entropic bias" that severely penalizes sharp localization. In tasks like crowd counting or density estimation, it causes predicted density maps to appear artificially blurred or overly concentrated around a global average, failing to resolve individual points.

2. Barycenter Blurring

A Wasserstein barycenter is the geometric average of multiple probability distributions. When computed using entropic OT, the regularization acts like a spatial low-pass filter.

In Debiased Sinkhorn barycenters, Hicham Janati et al. prove that this smoothing effect is an inherent structural property of the regularized objective rather than a numerical artifact. When averaging images or point clouds, the entropic penalty forces the barycenter to absorb a diffusion-like blur, washing out sharp edges, intricate textures, and fine structural details.

This issue is a recurring bottleneck when training generative models or aggregating high-dimensional data, as highlighted by Alexander Korotin et al. in Wasserstein Iterative Networks for Barycenter Estimation and the authors of Continuous Regularized Wasserstein Barycenters.

3. Map Collapse Toward the Mean

When estimating continuous transport maps (the actual functions that push one distribution onto another), entropic regularization introduces a severe centering bias.

In Debiaser Beware: Pitfalls of Centering Regularized Transport Maps, Aram-Alexandre Pooladian et al. show that entropic regularization shifts the estimated transport map toward the target distribution's mean. As the regularization strength $\varepsilon$ increases, the mapping loses its discriminative power and collapses toward a centralized average, failing to capture complex, non-linear relationships on the underlying data manifold.

The Algorithmic Tug-of-War

An obvious question arises: why not simply set $\varepsilon$ to a value very close to zero?

In practice, this triggers a brutal trade-off between statistical precision and numerical stability. As detailed by Lénaïc Chizat et al. in Faster Wasserstein Distance Estimation with the Sinkhorn Divergence, the Sinkhorn algorithm relies on scaling factors that scale exponentially with $-C_{ij}/\varepsilon$. When $\varepsilon$ approaches zero, these terms trigger severe numerical underflow and overflow, causing the algorithm to fail.

Furthermore, Mete Kemertas notes in Efficient and Accurate Optimal Transport with Mirror Descent and Conjugate Gradients that smaller values of $\varepsilon$ drastically slow down the convergence rate of iterative solvers. This creates a fundamental conflict: high regularization allows for fast, stable convergence but introduces massive bias, while low regularization preserves geometric detail at the cost of numerical failure and computational stagnation.

Mitigating the Bias

To break this deadlock, researchers have developed several strategies to eliminate entropic bias while preserving the computational benefits of regularized optimization.

Debiasing via Sinkhorn Divergences

One of the most widely adopted solutions is to mathematically subtract the self-transport bias. In Interpolating between Optimal Transport and MMD using Sinkhorn Divergences, Jean Feydy et al. introduced "Sinkhorn Divergences," which modify the regularized cost function by subtracting the self-transport terms:

$$W_\varepsilon(\alpha, \beta) = OT_\varepsilon(\alpha, \beta) - \frac{1}{2}OT_\varepsilon(\alpha, \alpha) - \frac{1}{2}OT_\varepsilon(\beta, \beta)$$

This formulation ensures that the distance from a distribution to itself is exactly zero, successfully interpolating between the strict geometry of classical OT and the computational efficiency of Maximum Mean Discrepancy (MMD). Hicham Janati et al. successfully adapted this debiasing formulation to barycenter problems, allowing for the recovery of perfectly sharp features without sacrificing the speed of standard Sinkhorn iterations.

Proximal Point and Annealing Schemes

Rather than solving a single, statically regularized problem, another approach uses entropic OT as a stepping stone to reach the exact, unregularized solution.

In A Fast Proximal Point Method for Computing Exact Wasserstein Distance, Yujia Xie et al. introduced the Inexact Proximal Point Method (IPOT). By maintaining entropic regularization inside a proximal loop and updating the execution matrix at each step, IPOT achieves linear convergence to the exact Wasserstein distance while remaining entirely robust to numerical instability.

Alternatively, researchers use dynamic annealing schemes that progressively shrink $\varepsilon$ over time. Ferdinand Genans et al. developed this concept in Decreasing Entropic Regularization Averaged Gradient for Semi-Discrete Optimal Transport, demonstrating that adaptively shrinking the regularization parameter eliminates the persistent smoothing bias while maintaining optimal convergence rates. Similarly, Marco Cuturi et al. leveraged progressive solvers in Progressive Entropic Optimal Transport Solvers to navigate the speed-accuracy trade-off safely without numerical collapse.

Alternative Optimization Geometries

Finally, some modern approaches bypass entropic scaling entirely by altering the optimization geometry. Marin Ballu et al. introduced Mirror Sinkhorn in Mirror Sinkhorn: Fast Online Optimization on Transport Polytopes, combining mirror descent with matrix scaling to optimize general convex functions over transport polytopes without regularization bias.

For continuous problems, Kaheon Kim et al. proposed alternating between Wasserstein and Sobolev ($\dot{\mathbb{H}}^1$) optimization geometries in Optimal Transport Barycenter via Nonconvex-Concave Minimax Optimization. By avoiding entropic smoothing altogether, their method preserves the structural integrity and visual fidelity of high-resolution, million-pixel images.

Key Foundational Work

A Fast Proximal Point Method for Computing Exact Wasserstein Distance (2019) — contributed 1 concept elements to this direction.
Interpolating between Optimal Transport and MMD using Sinkhorn Divergences (2019) — contributed 1 concept elements to this direction.
On the Convergence and Robustness of Training GANs with Regularized Optimal Transport (2018) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_entropic_regularization_bias_c3f78db7,
  title = {entropic regularization bias},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/entropic-regularization-bias/rdr_c3f78db7e2dd429da609f529f6855b16},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-19},
}

This exact version (version 1, frozen text):

@misc{lacuna_entropic_regularization_bias_c3f78db7_v1,
  title = {entropic regularization bias},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/entropic-regularization-bias/rdr_c3f78db7e2dd429da609f529f6855b16/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-19},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake