Asymptotic Guarantees for Generative Modeling Based on the Smooth Wasserstein Distance

Ziv Goldfeld, Kristjan H. Greenewald, Kengo KatoNeurIPS 2020 · OpenReview

TL;DR

Standard Wasserstein distances used in generative modeling suffer from the "curse of dimensionality," requiring exponentially more data as dimensions increase. This paper proves that by simply smoothing distributions with Gaussian noise, the resulting Smooth Wasserstein Distance (SWD) achieves a dimension-free convergence rate of $O(n^{-1/2})$. This provides the first rigorous high-dimensional statistical guarantees for Wasserstein-based generative models.

The Problem: The Curse of Dimensionality in MDE

Generative models like GANs and Wasserstein Autoencoders (WAEs) are often framed as Minimum Distance Estimation (MDE) problems. The goal is to find model parameters $\theta$ that minimize the distance between an empirical data distribution ($P_n$) and a parametric model ($Q_\theta$).

While the 1-Wasserstein distance ($W_1$) is a popular choice due to its stability and metric properties, it has a fatal flaw in high dimensions: the curse of dimensionality (CoD). The rate at which the empirical $W_1$ distance converges to the true distance is $O(n^{-1/d})$, where $d$ is the dimension. In a 100-dimensional space, this rate is practically zero, meaning theoretical guarantees based on standard $W_1$ are often meaningless for real-world, high-dimensional data.

Methods: The Smooth Wasserstein Distance

The authors propose using the Smooth 1-Wasserstein Distance (SWD), denoted as $W_1^{(\sigma)}$. This is defined as the standard $W_1$ distance between two distributions after they have both been convolved with isotropic Gaussian noise $\mathcal{N}(0, \sigma^2 I_d)$.

The paper provides a rigorous statistical treatment of the Minimum Smooth Wasserstein Estimator (MSWE): $$\hat{\theta}_n \in \text{argmin}_{\theta \in \Theta} W_1^{(\sigma)}(P_n, Q_\theta)$$

Key theoretical tools used to analyze this estimator include:

  • Empirical Process Theory: Proving that the class of functions associated with SWD is a "Donsker class," which allows for a uniform Central Limit Theorem (CLT).
  • Kantorovich-Rubinstein (KR) Duality: Used to represent the scaled empirical SWD as a supremum of an empirical process.
  • Asymptotic Analysis: Characterizing the limit distribution of both the optimal parameters $\hat{\theta}_n$ and the minimal distance itself.

Key Insights

  • Smoothing as a Regularizer: Gaussian smoothing "levels out" local irregularities in high-dimensional distributions. This doesn't just help with optimization; it fundamentally changes the statistical properties of the distance, bypassing the curse of dimensionality.
  • Dimension-Free Convergence: The most striking insight is that SWD converges at a rate of $O(n^{-1/2})$ regardless of the dimension $d$. This puts Wasserstein-based MDE on the same theoretical footing as classic maximum likelihood estimation.
  • High-Dimensional Limit Distributions: Unlike standard $W_1$, which only has a known limit distribution in 1D, the authors characterize a non-degenerate limit distribution for SWD in arbitrary dimensions. This allows for the use of the nonparametric bootstrap to estimate uncertainty in generative models.

Key Results

  • Generalization Bounds: The authors derive an $O(n^{-1/2})$ generalization bound for generative modeling based on MSWE that holds in any dimension.
  • Stability of Estimates: For a 1D mixture, the $\sqrt{n}$-scaled error in parameter estimation overlaps across different sample sizes $n$, confirming that a stable limit distribution exists as predicted by the theory.
1D limiting distributions for mean parameters and SWD convergence.
  • High-D Consistency: In dimensions 5 and 10, the distribution of the scaled variance estimate $\sqrt{n}(\hat{\sigma}_n^2 - \sigma_*^2)$ converges to a clear limit, demonstrating the estimator's reliability in higher dimensions.
Empirical limiting distributions for the variance parameter in d=5 and d=10.
  • Empirical Convergence Rate: Experiments demonstrate that even for a complex 10D mixture, the SWD loss converges at the predicted $O(n^{-1/2})$ rate, though smaller smoothing parameters ($\sigma$) require more samples for this rate to "kick in."

Limitations and Future Work

  • The Prefactor Problem: While the rate is dimension-free ($n^{-1/2}$), the constant factor (prefactor) in front of it still depends exponentially on $d$. The authors suggest exploring the manifold hypothesis (assuming data lies on a low-dimensional manifold) to alleviate this.
  • Computational Efficiency: The experiments used standard WGAN-GP style neural network estimators as placeholders. Developing efficient algorithms specifically tailored to exploit the Gaussian convolution structure of SWD is a key next step.
  • Noise Annealing: In practice, practitioners often "anneal" noise (start with high $\sigma$ and decrease it). The current theory assumes a fixed $\sigma$; analyzing the $\sigma \to 0$ regime remains a challenging theoretical frontier.

Summary updated

BibTeX
@inproceedings{goldfeld2020asymptotic,
  author = {Ziv Goldfeld and Kristjan H. Greenewald and Kengo Kato},
  title = {{Asymptotic Guarantees for Generative Modeling Based on the Smooth Wasserstein Distance}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2020},
  url = {https://openreview.net/forum?id=kykVBocZ0S},
}
Figure