BourGAN: Generative Networks with Metric Embeddings

Chang Xiao, Peilin Zhong, Changxi Zheng

Generative Adversarial Networks (GANs) are powerful tools for modeling complex data distributions, but they notoriously suffer from mode collapse. This occurs when the generator learns to produce only a limited subset of the variations present in the training data—for instance, a model trained on handwritten digits might only generate "1"s and "7"s, ignoring the rest.

BourGAN addresses this problem by rethinking the latent space. Instead of forcing a generator to map a single Gaussian distribution to a complex, multi-modal data manifold, BourGAN uses metric embedding theory to construct a Gaussian Mixture Model (GMM) in the latent space that mirrors the actual geometric structure of the data.

The Problem: Mode Collapse and Geometric Mismatch

The authors argue that mode collapse stems from a fundamental geometric mismatch. Standard GANs attempt to map a single, continuous multivariate Gaussian "blob" in the latent space to a data distribution that consists of many disconnected "islands" (modes).

Mapping a single continuous Gaussian to disconnected modes forces the generator to create unrealistic 'bridge' samples (red dots).

Because neural networks represent continuous functions, mapping a single continuous space to disconnected regions forces the generator into a dilemma: it must either collapse onto a single island or create "bridges" of unrealistic, low-quality samples between modes to maintain continuity. Furthermore, standard GAN architectures require the user to guess the "correct" dimensionality for the latent space, which can further hinder performance.

The BourGAN Methodology

BourGAN (short for Bourgain GAN) introduces a three-step pipeline to align the latent space with the data space:

  1. Metric Embedding (The Bourgain Theorem): The authors treat the dataset as a geometric structure in a metric space. They subsample the data and use Bourgain’s Theorem to embed these points into a low-dimensional $\ell_2$ space. This mathematical embedding is guaranteed to preserve the pairwise distance distribution of the original data.
  2. Automatic Latent Space Construction: Unlike standard GANs, BourGAN automatically determines the optimal latent space dimensionality based on the embedding theory ($k = O(\log m)$). It then constructs a Gaussian Mixture Model where each component is centered at one of the embedded points, naturally representing the data's distinct modes.
  3. Distance-Preserving Objective: During training, the standard GAN loss is augmented with a new term ($L_{dist}$). This loss encourages the generator to ensure that the distance between two generated samples in the data space matches the distance between their corresponding latent vectors. The authors utilize logarithmic distances to ensure the optimization isn't dominated by outliers.

Key Insights and Metric Flexibility

By using metric embeddings, the latent space is no longer a "black box"; it becomes a structured map of the data. If two points are far apart in the data metric, they are far apart in the latent GMM. This approach solves the continuity problem: because the latent space is already divided into a GMM, the generator does not suffer from the arbitrarily large gradients that make standard GAN training unstable.

BourGAN allows the latent space to reflect distinct data modes, such as the 10 digit classes in MNIST, based on the chosen distance metric.

Crucially, BourGAN offers metric flexibility. It allows users to define what a "mode" is by choosing a specific distance metric. For example, using a "classifier distance" on MNIST allows the model to explicitly recognize 10 distinct digit modes, whereas a simple pixel-wise $\ell_2$ distance might only perceive two broad clusters.

Empirical Results

The authors evaluated BourGAN across several benchmarks, demonstrating significant improvements over existing methods:

  • Superior Mode Capture: On a synthetic 2D Grid dataset with 25 modes, BourGAN captured all 25 modes with high precision. Competing models like Unrolled GANs and VEEGAN either struggled with sample quality or missed modes entirely.
  • Avoiding "Unwanted Samples": In a 2D Ring test, BourGAN produced almost zero "garbage" samples in the empty space between modes, whereas competitive models frequently generated unrealistic bridges.
  • Stacked MNIST Performance: On Stacked MNIST—a notoriously difficult dataset with 1,000 possible modes designed specifically to test mode collapse—BourGAN captured significantly more modes than previous state-of-the-art methods, maintaining a much higher entropy in the generated distribution.
  • Robust Stability: The method proved highly robust to challenging geometries, such as a "2D Circle" dataset where a central mode is often ignored by other GAN variants.

Limitations and Future Directions

While highly effective, BourGAN does have limitations. The embedding process requires computing pairwise distances for a subsample of the data. While the authors show that a subsample of $m=4096$ is sufficient for their tests, this $O(m^2)$ step could become a computational bottleneck for extremely complex or massive datasets if a larger $m$ is required. Additionally, the quality of the captured modes is heavily dependent on the user-chosen distance metric; if the metric fails to capture the semantic essence of the data, the latent GMM will reflect that failure.

Looking forward, the authors suggest that this metric embedding approach is not strictly limited to GANs. It could be integrated into other generative frameworks, such as Variational Autoencoders (VAEs), to fundamentally improve how latent spaces are structured and utilized.

Summary updated

Abstract

This paper addresses the mode collapse for generative adversarial networks (GANs). We view modes as a geometric structure of data distribution in a metric space. Under this geometric lens, we embed subsamples of the dataset from an arbitrary metric space into the L2 space, while preserving their pairwise distance distribution. Not only does this metric embedding determine the dimensionality of the latent space automatically, it also enables us to construct a mixture of Gaussians to draw latent space random vectors. We use the Gaussian mixture model in tandem with a simple augmentation of the objective function to train GANs. Every major step of our method is supported by theoretical analysis, and our experiments on real and synthetic data confirm that the generator is able to produce samples spreading over most of the modes while avoiding unwanted samples, outperforming several recent GAN variants on a number of metrics and offering new features.
BibTeX
@inproceedings{xiao2018bourgan,
  author = {Chang Xiao and Peilin Zhong and Changxi Zheng},
  title = {{BourGAN: Generative Networks with Metric Embeddings}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2018},
  url = {https://openreview.net/forum?id=rJEMew-uWH},
}
Figure