Geometrical Insights for Implicit Generative Modeling

Léon Bottou, Martín Arjovsky, David Lopez-Paz, Maxime Oquab

This paper, authored by a team at Facebook AI Research (including deep learning pioneer Leon Bottou), provides a rigorous mathematical explanation for a phenomenon many practitioners have observed: why the Wasserstein distance often works better than other metrics (like MMD or Energy Distance) for training GANs, despite having much worse statistical properties.

TL;DR

The paper argues that the success of implicit generative models (like GANs) depends on the "geometrical fit" between the loss function and the generator's architecture. While Wasserstein distance is harder to estimate from samples, its "displacement" geometry perfectly matches how neural networks transform distributions, leading to fewer local minima compared to "mixture-based" distances like MMD.

What Problem Does This Paper Address?

Implicit generative models don't define a probability density function; instead, they learn a function $G_\theta$ that transforms simple noise $z$ into complex data $x$. To train these, we need a way to measure the "distance" between the generated distribution $P_\theta$ and the real data distribution $Q$.

Common choices include the Wasserstein Distance (WD), Energy Distance (ED), and Maximum Mean Discrepancy (MMD). Theoretically, ED and MMD are attractive because they are easy to estimate from finite samples. In contrast, WD has a "curse of dimensionality"—it requires a massive amount of data to estimate accurately. Yet, in practice, Wasserstein GANs often produce much better results. The authors set out to explain this paradox using the lens of geodesic geometry.

Methods

The authors analyze these distances not just as numbers, but as "length spaces" that define how one distribution can be deformed into another along a path (a geodesic). They distinguish between two types of "convexity":

  1. Mixture Convexity: Related to "blending" distributions (e.g., $0.5 \cdot \text{Cat} + 0.5 \cdot \text{Dog}$). This is the natural geometry of ED and MMD.
  2. Displacement Convexity: Related to "moving" mass (e.g., physically shifting the pixels of a cat image until it looks like a dog). This is the natural geometry of Wasserstein distances.

The paper uses Theorem 6.3 to show that if both the model family and the loss function are convex with respect to the same type of geometry, gradient descent is guaranteed to reach a global optimum.

Datasets

As a theoretical paper, the focus is on mathematical proofs and toy examples. However, for qualitative validation, the authors use:

  • LSUN Bedrooms: A standard large-scale dataset of bedroom images used to compare the visual quality of models trained with ED versus WGAN-GP (Wasserstein).
  • Synthetic 2D Distributions: Toy examples (like line segments in $\mathbb{R}^2$) are used to visualize level sets and demonstrate the existence of local minima.

Key Insights and Ideas

  • The Architecture-Loss Mismatch: Neural network generators are "displacement engines." Changing a weight $\theta$ slightly moves the generated points in space; it doesn't "fade" one distribution into another. Because ED/MMD rely on mixture geometry, they create "spurious local minima" when paired with these displacement-based generators.
  • Wasserstein's "Aha" Moment: The 1-Wasserstein distance ($W_1$) is unique because it is compatible with both mixture and displacement geometries. This makes it incredibly robust.
  • Approximate Global Convergence: The authors prove that for $W_1$, the "level sets" of the loss function are connected (Theorem 6.9). This means that even if the neural network is non-convex, the optimization landscape for $W_1$ is much friendlier, allowing gradient descent to find its way to a good solution without getting stuck in high-cost local minima.

Key Results

  • Qualitative Superiority: Figure 2 in the paper clearly shows that a model trained with the Wasserstein distance produces recognizable bedroom images, while the same architecture trained with Energy Distance produces blurry, incoherent blobs.
  • Theoretical Guarantee: They establish that for the 1-Wasserstein distance, optimization will not stop until the model is within a specific distance (related to the diameter of the data) of the global minimum. This holds even for highly non-convex neural network parametrizations.

Limitations and Future Work

  • The Statistical Gap: The paper acknowledges that while $W_1$ has superior optimization geometry, it still suffers from poor sample complexity. A major open question is how to combine the "good geometry" of Wasserstein with the "good statistics" of MMD.
  • Coarse Bounds: The convergence guarantees provided are "coarse"—they ensure you get close to a global minimum, but they don't guarantee you'll hit it exactly.
  • Generator Capacity: The proofs assume the generator has enough capacity to follow the displacement geodesics. In practice, the restricted architecture of a specific neural network might still introduce its own local minima.

The authors suggest that future research should look for "hybrid" distances that capture the best of both worlds, potentially leading to even more stable and data-efficient generative models.

Summary updated

Abstract

Learning algorithms for implicit generative models can optimize a variety of criteria that measure how the data distribution differs from the implicit model distribution, including the Wasserstein distance, the Energy distance, and the Maximum Mean Discrepancy criterion. A careful look at the geometries induced by these distances on the space of probability measures reveals interesting differences. In particular, we can establish surprising approximate global convergence guarantees for the 1-Wasserstein distance, even when the parametric generator has a nonconvex parametrization.
BibTeX
@inproceedings{bottou2017geometrical,
  author = {Léon Bottou and Martín Arjovsky and David Lopez-Paz and Maxime Oquab},
  title = {{Geometrical Insights for Implicit Generative Modeling}},
  booktitle = {Braverman Readings in Machine Learning},
  year = {2017},
  url = {https://openreview.net/forum?id=YP9KecpFkE},
}
Figure