Geometrical Insights for Implicit Generative Modeling

Léon Bottou, Martín Arjovsky, David Lopez-Paz, Maxime Oquab

Introduction

Implicit generative models, such as Generative Adversarial Networks (GANs), have revolutionized unsupervised learning by learning to transform simple noise into complex data distributions. A central challenge in training these models is selecting a distance metric to compare the generated distribution $P_\theta$ with the real data distribution $Q$.

While metrics like Maximum Mean Discrepancy (MMD) and Energy Distance (ED) are statistically attractive due to their ease of estimation from finite samples, the Wasserstein distance (WD) often performs better in practice. This paper, authored by a team at Facebook AI Research, explains this paradox through the lens of geodesic geometry, arguing that the success of GANs depends on the "geometrical fit" between the loss function and the generator's architecture.

Mixture vs. Displacement Geometry

The authors distinguish between two types of convexity that define how distributions can be deformed into one another:

  1. Mixture Convexity: Associated with "blending" distributions (e.g., $0.5 \cdot \text{Cat} + 0.5 \cdot \text{Dog}$). This is the natural geometry for distances like MMD, ED, and standard $f$-divergences.
  2. Displacement Convexity: Associated with "moving" mass (e.g., shifting pixels to transform a cat image into a dog). This is the natural geometry of Wasserstein distances.
Figure 1: Visualization of displacement geometry where distributions $P_\theta$ shift along the parameter space $\theta$.

Neural network generators are essentially "displacement engines." Adjusting a weight $\theta$ moves generated points in space rather than fading between distributions. When a displacement-based generator is paired with a mixture-based loss, it often results in "spurious local minima."

The Advantage of Wasserstein Distance

The 1-Wasserstein distance ($W_1$) is uniquely robust because it is compatible with both mixture and displacement geometries. This compatibility ensures a friendlier optimization landscape. The paper provides theoretical guarantees, such as Theorem 6.3, showing that gradient descent can reach a global optimum if the model family and loss function share the same geometric convexity.

Furthermore, the authors establish that for $W_1$, the level sets of the loss function are connected (Theorem 6.9), allowing optimization to find good solutions even in the presence of non-convex neural network parametrizations. In contrast, standard $f$-divergences (summarized in the table below) often struggle with these displacement-based architectures.

Table 1: Common f-divergences and their conjugate functions, which typically rely on mixture-based geometry.

Key Results and Qualitative Validation

Qualitative experiments on the LSUN Bedrooms dataset highlight the impact of this geometric mismatch. Models trained with Wasserstein distance produce recognizable, high-quality images, while those trained with Energy Distance result in blurry, incoherent outputs. This suggests that the optimization benefits of Wasserstein distance outweigh its statistical estimation challenges.

Conclusion and Future Work

The paper concludes that while $W_1$ has superior optimization geometry, its poor sample complexity remains a hurdle. Future research should focus on developing "hybrid" distances that capture the optimization benefits of displacement geometry while maintaining the statistical efficiency of mixture-based metrics like MMD. This could lead to more stable and data-efficient generative models.

Summary updated

BibTeX
@inproceedings{bottou2017geometrical,
  author = {Léon Bottou and Martín Arjovsky and David Lopez-Paz and Maxime Oquab},
  title = {{Geometrical Insights for Implicit Generative Modeling}},
  booktitle = {Braverman Readings in Machine Learning},
  year = {2017},
  url = {https://openreview.net/forum?id=YP9KecpFkE},
}
Figure