Dimension Independent Statistical Convergence for Probability Distances

This direction is merged into another direction: dimension-independent convergence (dimension-independent sample complexity). Its last version stays readable and citable here.

Version 1 of 154 papers65 conceptsfrozen 2026-10-04

In high-dimensional machine learning, we often need to measure how "close" two probability distributions are. This task is central to training Generative Adversarial Networks (GANs), performing two-sample tests, and conducting Bayesian inference. However, traditional metrics like the Wasserstein distance suffer from a "curse of dimensionality": the number of samples required to accurately estimate the distance grows exponentially with the number of dimensions.

A growing body of research has identified a family of metrics—including Sliced Wasserstein distances, Maximum Mean Discrepancy (MMD), and smoothed optimal transport—that achieve dimension-independent statistical convergence. These metrics allow us to compare complex, high-dimensional data with a sample complexity that scales based on the number of observations rather than the ambient dimensionality of the space.

The Curse of Dimensionality in Optimal Transport

To understand why dimension-independent metrics are a breakthrough, we must first look at the failure of standard Optimal Transport (OT). As Ugo Tanielian, Gérard Biau, and Maxime Sangnier explain in Some Theoretical Insights into Wasserstein GANs, the empirical Wasserstein distance typically converges to its population counterpart at a rate of $O(n^{-1/d})$. In a 100-dimensional space, this rate is so slow that the distance becomes statistically meaningless without an astronomical amount of data.

The "curse" arises because the Wasserstein distance tries to account for every possible geometric detail in the high-dimensional space. To bypass this, researchers have turned to mechanisms that simplify the geometry or restrict the class of functions used to measure the distance.

Slicing: Reducing Complexity via Projections

One of the most successful strategies for achieving dimension-independence is "slicing." The core intuition is that while comparing distributions in $d$ dimensions is hard, comparing them in 1 dimension is easy.

Khai Nguyen provides a foundation for this in An Introduction to Sliced Optimal Transport, showing that Sliced Wasserstein (SW) distances decompose a high-dimensional problem into an infinite collection of 1D Wasserstein problems. Because 1D distances converge at a parametric rate of $O(n^{-1/2})$, the integrated average of these "slices" also converges at a rate independent of the ambient dimension.

Recent work has refined this by learning which "slices" are most informative. Ruben Ohana, Kimia Nadjahi, and colleagues in Shedding a PAC-Bayesian Light on Adaptive Sliced-Wasserstein Distances use PAC-Bayesian theory to prove that even when we adaptively choose projection directions based on the data, the convergence remains dimension-free. Similarly, Sloan Nietert and Ziv Goldfeld demonstrate in Statistical, Robustness, and Computational Guarantees for Sliced Wasserstein Distances that for log-concave distributions, the error scales at a rate that avoids the $1/d$ exponent entirely.

Kernel Methods and Maximum Mean Discrepancy

Another path to dimension-independence lies in Reproducing Kernel Hilbert Spaces (RKHS). Instead of comparing distributions directly in the input space, we map them to an infinite-dimensional feature space using a kernel.

The Maximum Mean Discrepancy (MMD) is a prime example. As noted by Alessandro Barp and Mark Girolami in Statistical Inference for Generative Models with Maximum Mean Discrepancy, MMD provides a convergence rate for distribution matching that is independent of the data's dimensionality. This property makes it a scalable alternative for training generative models where OT would fail.

The reason MMD works is that the "smoothness" of the kernel effectively limits the complexity of the functions used to distinguish the distributions. Minh Ha Quang formalizes this in Kullback-Leibler and Renyi divergences in reproducing kernel Hilbert space and Gaussian process settings, proving that divergence estimation in these functional spaces achieves sample complexity that does not scale with the dimensionality of the underlying feature space.

Smoothing as a Regularizer

A third mechanism for breaking the curse is Gaussian smoothing. By convolving probability distributions with a small amount of Gaussian noise, we "blur" the fine-grained geometric details that cause the $O(n^{-1/d})$ slow-down.

In Smooth p-Wasserstein Distance: Structure, Empirical Approximation, and Statistical Applications, researchers show that applying a Gaussian kernel allows the resulting Wasserstein distance to achieve a parametric convergence rate of $n^{-1/2}$ regardless of dimension. This idea is extended to the privacy-preserving context by Mokhtar Z. Alaya and Alain Rakotomamonjy in Gaussian-Smoothed Sliced Probability Divergences, who prove that adding noise for privacy does not destroy the statistical efficiency of the underlying slicing technique.

Why It Works: The Role of Structure

The common thread across these papers is that dimension-independence is not magic; it is the result of assuming or imposing structure.

  1. Low-Dimensional Projections: Sliced distances assume that the difference between two high-dimensional distributions can be captured by looking at their 1D projections.
  2. Functional Smoothness: Kernel methods and smoothing assume that we only care about "smooth" differences between distributions, ignoring high-frequency noise that is difficult to estimate in high dimensions.
  3. Local Regularity: As Bertrand Michel and Frédéric Chazal show in Rates of convergence for robust geometric inference, if the data lies on a low-dimensional manifold or satisfies certain local regularity conditions, geometric inference can achieve a $1/\sqrt{n}$ rate even if the ambient space is massive.

By moving away from "agnostic" metrics like the standard Wasserstein distance and toward metrics that leverage these structural properties, the field has developed tools that remain statistically powerful even as datasets grow in complexity and dimensionality.

Papers in this version

History & citationVersion 1 · 2026-10-04

Merged into another direction · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_dimension_independent_statistical_conver_9464a727,
  title = {Dimension Independent Statistical Convergence for Probability Distances},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/dimension-independent-statistical-convergence-for-probability-distances/rdr_9464a7271b3140beb87eb0e28fa8f7f5},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_dimension_independent_statistical_conver_9464a727_v1,
  title = {Dimension Independent Statistical Convergence for Probability Distances},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/dimension-independent-statistical-convergence-for-probability-distances/rdr_9464a7271b3140beb87eb0e28fa8f7f5/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}