Probability distributions are the core currency of modern machine learning, appearing everywhere from generative modeling to shape analysis. To compare these distributions, Optimal Transport (OT) has emerged as a gold standard. By computing the minimum cost required to morph one distribution into another, OT captures the underlying geometry of the data space.
However, exact optimal transport comes with a steep computational price: solving it for $n$ data points requires solving a linear program with $O(n^3 \log n)$ complexity. To scale this up, researchers widely rely on entropic regularization. By adding an entropy penalty to the transport plan, the problem becomes strictly convex and can be solved in $O(n^2)$ time using the elegant, GPU-friendly Sinkhorn algorithm.
But this computational speedup introduces a severe side effect: entropic bias.
The Cost of Speed: Entropic Blurring
To understand entropic bias, we must look at what the entropy penalty actually does. The standard entropic optimal transport objective is formulated as:
$$\text{OT}_\epsilon(\mu, \nu) = \min_{P \in \Pi(\mu, \nu)} \langle P, C \rangle - \epsilon H(P)$$
Here, $C$ is the cost matrix, $P$ is the transport plan, and $H(P)$ is the Shannon entropy of the plan. The regularization parameter $\epsilon$ controls the trade-off between geometric fidelity and smoothness.
When $\epsilon > 0$, the objective actively rewards transport plans with higher entropy—meaning plans that are diffuse, spread out, and blurry. This creates a fundamental mathematical paradox: the regularized distance from a distribution to itself is no longer zero.
$$\text{OT}_\epsilon(\mu, \mu) > 0$$
Because a perfectly deterministic identity mapping has zero entropy, the regularizer forces the transport plan to leak mass to nearby points just to maximize entropy. This violation of the identity of indiscernibles means that $\text{OT}_\epsilon$ is no longer a true metric.
In practice, this entropic bias acts like an artificial force field or a "Gaussian blur" during optimization:
- In Generative Models (GANs/VAEs): The loss function rewards high-entropy, diffuse states, causing the model to generate blurry samples.
- In Barycenter Estimation: Averaging multiple distributions results in washed-out, over-smoothed "smudges" instead of sharp, localized features.
- In Shape Registration: Shapes artificially shrink or lose mass toward their centroids because the metric penalizes the sharp boundaries required to match them exactly.
The Fix: Self-Transport Subtraction
To eliminate this artifact while preserving the computational speed of the Sinkhorn algorithm, a powerful idea emerged: debiasing via self-transport subtraction.
The core mechanism defines a new objective, known as the Sinkhorn Divergence:
$$S_\epsilon(\mu, \nu) = \text{OT}_\epsilon(\mu, \nu) - \frac{1}{2}\text{OT}_\epsilon(\mu, \mu) - \frac{1}{2}\text{OT}_\epsilon(\nu, \nu)$$
The intuition behind this formula is beautifully simple. Think of $\text{OT}_\epsilon(\mu, \mu)$ as a measure of the "auto-correlation" or the intrinsic entropic blur of the distribution $\mu$ itself. By subtracting the self-transport costs of both the source and target distributions from their cross-transport cost, we cancel out the first-order geometric distortions introduced by the entropy term.
As proved by Jean Feydy et al. in "Interpolating between Optimal Transport and MMD using Sinkhorn Divergences", this algebraic correction completely restores the metric properties of the objective. The resulting Sinkhorn Divergence is strictly positive, convex, and—crucially—evaluates to exactly zero if and only if $\mu = \nu$.
Furthermore, Lénaïc Chizat et al. demonstrated in Faster Wasserstein Distance Estimation with the Sinkhorn Divergence that this self-correction term cancels out the primary asymptotic bias relative to the true, unregularized Wasserstein distance. This allows practitioners to use much larger regularization parameters $\epsilon$ (leading to faster convergence) without suffering from the typical geometric distortion.
Restoring Sharpness Across Domains
By ensuring that a distribution's distance to itself is zero, debiased entropic transport changes the behavior of optimization gradients, preventing distributions from collapsing or blurring.
Sharp Generative Modeling
When training generative models, the objective function serves as the guide for the generator. In Learning Generative Models with Sinkhorn Divergences, Aude Genevay et al. showed that shifting from standard entropic OT to the debiased formulation allows models to accurately recover the true data distribution instead of a smoothed approximation.
Similarly, Maziar Sanjabi et al. in On the Convergence and Robustness of Training GANs with Regularized Optimal Transport and Giorgio Patrini et al. in Sinkhorn AutoEncoders leveraged this debiasing effect to eliminate the characteristic blurriness of regularized objectives, enabling the generation of sharp, high-fidelity samples while retaining a smooth loss landscape.
Crisp Barycenters and Shape Registration
Computing the average of multiple distributions under standard entropic transport results in a diffuse cloud. In Debiased Sinkhorn barycenters, Hicham Janati et al. showed that subtracting the self-transport terms completely neutralizes this diffusion effect. This allows the algorithm to compute sharp, clean structural averages at the same high speed as standard Sinkhorn iterations.
Standard Entropic Barycenter: [ Blurry / Diffuse Smudge ]
Debiased Sinkhorn Barycenter: [ Sharp / Well-Defined Shape ]
This property is equally vital when aligning complex geometries. Lucas de Lara et al. demonstrated in Diffeomorphic Registration using Sinkhorn Divergences that debiasing prevents shapes from artificially shrinking or losing mass during registration, ensuring that the computed transformations accurately map target boundaries.
Generalizing the Debiasing Principle
The core insight of self-similarity normalization has been successfully extended far beyond standard optimal transport:
- Time Series Alignment: Standard Dynamic Time Warping (DTW) can be smoothed to create a differentiable alignment loss, but it suffers from a similar entropic bias. In Differentiable Divergences Between Time Series, Mathieu Blondel et al. introduced the soft-DTW divergence by subtracting self-alignment terms, ensuring that a sequence's distance to itself is zero and preventing temporal smoothing artifacts.
- Crowd Counting: Instead of using blurry Gaussian density maps to estimate crowd sizes, Hui LIN et al. in Direct Measure Matching for Crowd Counting utilized a semi-balanced Sinkhorn divergence. The debiasing term restores the identity of indiscernibles, forcing the model to produce sharp, well-localized peaks at exact person locations.
- Relational and Low-Rank Structures: The debiasing framework also applies to relational matching problems like the Gromov-Wasserstein distance (Guillaume Houry et al. in Gromov-Wasserstein at Scale, Beyond Squared Norms) and low-rank transport approximations (Meyer Scetbon et al. in Low-rank Optimal Transport: Approximation, Statistics and Debiasing), where it prevents gradient flows from collapsing into simplified low-rank skeletons.
- Stochastic Mini-Batching: When scaling OT to massive datasets using random mini-batches, a positive bias is introduced because a random batch is never perfectly identical to itself. Khai Nguyen et al. addressed this in On Transportation of Mini-batches: A Hierarchical Approach by using a hierarchical coupling that restores the identity property and eliminates mini-batch estimation bias.
Alternative Perspectives and Pitfalls
While self-transport subtraction is highly effective, it is not the only way to eliminate entropic bias, nor is it a universal free lunch.
Iterative Refinement (IPOT)
An alternative to changing the objective function is to change the optimization method itself. In A Fast Proximal Point Method for Computing Exact Wasserstein Distance, Yujia Xie et al. proposed the Inexact Proximal point method for exact Optimal Transport (IPOT). Instead of using a static entropy penalty, IPOT solves a sequence of regularized sub-problems where each step is regularized relative to the previous iteration's solution. This iterative refinement allows the algorithm to converge to the exact, unbiased Wasserstein distance while maintaining the computational robustness and scaling of Sinkhorn methods.
The Map Estimation Pitfall
Practitioners must also distinguish between debiasing a distance value and debiasing a transport map. In Debiaser Beware: Pitfalls of Centering Regularized Transport Maps, Aram-Alexandre Pooladian et al. demonstrated that while self-potential subtraction perfectly corrects distance metrics, applying the same centering technique to estimate the actual point-to-point transport maps can backfire. In finite-sample regimes, the subtraction step can inject significant statistical noise, meaning that standard entropic plans can sometimes outperform debiased ones when mapping actual data samples.
Summary
The development of debiased entropic optimal transport represents a major milestone in geometric machine learning. By recognizing that the smoothing effect of entropic regularization could be quantified and subtracted out via self-transport terms, researchers successfully combined the geometric sensitivity of classical optimal transport with the speed of modern regularized algorithms.
Key Foundational Work
Interpolating between Optimal Transport and MMD using Sinkhorn Divergences (2019) — contributed 2 concept elements to this direction.
Learning Generative Models with Sinkhorn Divergences (2018) — contributed 2 concept elements to this direction.
On the Convergence and Robustness of Training GANs with Regularized Optimal Transport (2018) — contributed 2 concept elements to this direction.