Abstract
Discrepancy measures between probability distributions, often termed statistical distances, are ubiquitous in probability theory, statistics and machine learning. To combat the curse of dimensionality when estimating these distances from data, recent work has proposed smoothing out local irregularities in the measured distributions via convolution with a Gaussian kernel. Motivated by the scalability of this framework to high dimensions, we investigate the structural and statistical behavior of the Gaussian-smoothed $p$-Wasserstein distance $\mathsf{W}_p^{(\sigma)}$, for arbitrary $p\geq 1$. After establishing basic metric and topological properties of $\mathsf{W}_p^{(\sigma)}$, we explore the asymptotic statistical behavior of $\mathsf{W}_p^{(\sigma)}(\hat\mu_n,\mu)$, where $\hat\mu_n$ is the empirical distribution of $n$ independent observations from $\mu$. We prove that $\mathsf{W}_p^{(\sigma)}$ enjoys a parametric empirical convergence rate of $n^{-1/2}$, which contrasts the $n^{-1/d}$ rate for unsmoothed $\mathsf{W}_p$ when $d \geq 3$. Our proof relies on controlling $\mathsf{W}_p^{(\sigma)}$ by a $p$th-order smooth Sobolev distance $\mathsf{d}_p^{(\sigma)}$ and deriving the limit distribution of $\sqrt{n}\,\mathsf{d}_p^{(\sigma)}(\hat\mu_n,\mu)$, for all dimensions $d$. As applications, we provide asymptotic guarantees for two-sample testing and minimum distance estimation using $\mathsf{W}_p^{(\sigma)}$, with experiments for $p=2$ using a maximum mean discrepancy formulation of $\mathsf{d}_2^{(\sigma)}$.
Introduction
The Wasserstein distance ($W_p$) has become a cornerstone of modern machine learning, particularly in generative modeling and domain adaptation, due to its ability to capture the underlying geometry of probability distributions. However, $W_p$ suffers from a significant theoretical bottleneck: the curse of dimensionality. When estimating the distance between two high-dimensional distributions from $n$ samples, the error typically scales as $n^{-1/d}$. In a 100-dimensional space, the number of samples required for a reliable estimate becomes astronomical.
This research investigates a "smoothed" version of the distance, the Gaussian-smoothed $p$-Wasserstein distance ($W_p^{(\sigma)}$), which bypasses this curse. By convolving distributions with a Gaussian kernel $\mathcal{N}_\sigma$, the estimation error drops to a parametric rate of $n^{-1/2}$, independent of the ambient dimension $d$.
Methodology
The authors establish the structural and statistical properties of $W_p^{(\sigma)}$ for arbitrary $p \geq 1$. The core approach involves:
- Metric and Topological Analysis: Proving that $W_p^{(\sigma)}$ is a valid metric that preserves the topology of the original $W_p$.
- Sobolev Distance Bridge: Introducing a $p$th-order smooth Sobolev Integral Probability Metric (IPM), denoted as $\mathsf{d}_p^{(\sigma)}$, which serves as an upper bound for $W_p^{(\sigma)}$.
- Empirical Process Theory: Utilizing the $\mu$-Donsker property of the Sobolev function class to derive parametric convergence rates and limit distributions.
- MMD Formulation: For $p=2$, showing that $\mathsf{d}_2^{(\sigma)}$ is equivalent to a Maximum Mean Discrepancy (MMD) with a specific closed-form kernel.
Experiments were conducted on synthetic datasets, including uniform distributions on hypercubes and Gaussian mixtures, to validate these theoretical findings across dimensions $d \in \{1, 3, 5, 6, 10\}$.
Key Results
The primary contribution is the proof of a dimension-independent parametric convergence rate. As shown in Figure 2, the empirical smoothed Wasserstein distance $W_2^{(\sigma)}(\hat{\mu}_n, \mu)$ converges at a rate of $n^{-1/2}$ across various dimensions and smoothing parameters. This is a massive improvement over the $n^{-1/d}$ rate of the unsmoothed distance.
Furthermore, the authors derived the limit distributions for the empirical smooth Sobolev distance. This theoretical foundation enables statistically sound two-sample testing. Figure 4 illustrates the density estimates of the scaled Sobolev distance, showing convergence to a stable limit distribution as the sample size $n$ increases, regardless of the smoothing level.
Beyond testing, the framework was applied to "Minimum Smooth Wasserstein Estimation" (M-SWE). Experiments confirmed that M-SWE is consistent and follows the fast $n^{-1/2}$ convergence rate in practice, making it a viable tool for high-dimensional generative modeling.
Discussion and Future Work
While Gaussian smoothing successfully breaks the curse of dimensionality in terms of the rate of convergence, the constant factors involved may still scale exponentially with dimension. This suggests that very high-dimensional problems remain challenging. Future research could investigate whether these rates depend on the intrinsic dimension of the data manifold rather than the ambient space. Additionally, the computational efficiency of $W_p^{(\sigma)}$ for $p \neq 2$ and the potential links between smoothing and differential privacy are promising avenues for further exploration.
Summary updated