BourGAN: Generative Networks with Metric Embeddings

Chang Xiao, Peilin Zhong, Changxi Zheng

Abstract

This paper addresses the mode collapse for generative adversarial networks (GANs). We view modes as a geometric structure of data distribution in a metric space. Under this geometric lens, we embed subsamples of the dataset from an arbitrary metric space into the l2 space, while preserving their pairwise distance distribution. Not only does this metric embedding determine the dimensionality of the latent space automatically, it also enables us to construct a mixture of Gaussians to draw latent space random vectors. We use the Gaussian mixture model in tandem with a simple augmentation of the objective function to train GANs. Every major step of our method is supported by theoretical analysis, and our experiments on real and synthetic data confirm that the generator is able to produce samples spreading over most of the modes while avoiding unwanted samples, outperforming several recent GAN variants on a number of metrics and offering new features.

Introduction: The Challenge of Mode Collapse

Generative Adversarial Networks (GANs) are powerful generative models but frequently suffer from mode collapse, where the generator learns only a subset of the training data's variations. The authors identify a fundamental geometric mismatch as the root cause: standard GANs attempt to map a single, unimodal Gaussian distribution in the latent space to a complex, multi-modal distribution in the data space.

As illustrated in Figure 1, this forces the generator to "stretch" its mapping across low-density regions between data modes. This leads to unstable training and the production of "unwanted" samples—blurry or unrealistic images that exist in the gaps between actual data clusters.

Comparison of latent space mappings: (a, b) standard GANs mapping a single Gaussian to multiple modes, causing 'unwanted' samples; (c) BourGAN's GMM latent space matching the data topology.

Methodology: Geometric Metric Embeddings

BourGAN addresses this by ensuring the latent space mirrors the geometric structure of the data space before training begins.

1. Metric Embeddings and Bourgain’s Theorem

The framework treats data modes as a geometric structure defined by a distance metric. Using Bourgain’s Theorem, a representative subsample of the dataset is embedded into a low-dimensional $\ell_2$ space. This embedding is mathematically guaranteed to preserve the pairwise distance relationships of the original data.

2. Latent Gaussian Mixture Model (GMM)

Instead of a single Gaussian, BourGAN constructs a Gaussian Mixture Model in the latent space, centered at the embedded points. This ensures that the latent space has "peaks" that correspond directly to the "peaks" (modes) in the real data, making the generator's mapping task well-posed.

3. Automatic Dimensionality and Distance-Preserving Loss

The embedding theory automatically determines the required latent dimensionality ($k = O(\log m)$) based on data complexity. To maintain this structure during training, the standard GAN objective is augmented with a distance-preserving loss term ($L_{dist}$), which encourages the generator to match the log-pairwise distances of generated samples to their corresponding latent vectors.

Experimental Results

BourGAN was evaluated on synthetic 2D distributions and standard benchmarks like MNIST, Stacked MNIST, and CelebA.

Mode Coverage and Sample Quality

On the Stacked MNIST benchmark, which features 1,000 distinct modes, BourGAN successfully captured all 1,000 modes, significantly outperforming standard DCGAN architectures. In synthetic tests, BourGAN was uniquely capable of producing clean, distinct modes without the "bridge" samples (unwanted artifacts) common in other models.

t-SNE visualizations and distance distributions showing BourGAN's ability to capture distinct, well-separated modes in the data compared to baseline approaches.

Convergence and Flexibility

The structured latent space and a brief supervised pre-training step allow BourGAN to stabilize and converge much faster than competitors like VEEGAN or PacGAN. Furthermore, the framework is flexible; by changing the distance metric (e.g., using feature-space distances from a pre-trained classifier), BourGAN can be tuned to discover modes based on semantic content rather than raw pixel values.

Conclusion

BourGAN provides a robust solution to mode collapse by rethinking the latent space through a geometric lens. By aligning the topology of the latent space with that of the data space using metric embeddings, it enables stable training and high-fidelity generation across all data modes. While the initial embedding step introduces some computational overhead, the resulting improvements in mode coverage and training stability offer a significant advancement for generative modeling.

Summary updated

Abstract

This paper addresses the mode collapse for generative adversarial networks (GANs). We view modes as a geometric structure of data distribution in a metric space. Under this geometric lens, we embed subsamples of the dataset from an arbitrary metric space into the l2 space, while preserving their pairwise distance distribution. Not only does this metric embedding determine the dimensionality of the latent space automatically, it also enables us to construct a mixture of Gaussians to draw latent space random vectors. We use the Gaussian mixture model in tandem with a simple augmentation of the objective function to train GANs. Every major step of our method is supported by theoretical analysis, and our experiments on real and synthetic data confirm that the generator is able to produce samples spreading over most of the modes while avoiding unwanted samples, outperforming several recent GAN variants on a number of metrics and offering new features.
BibTeX
@inproceedings{xiao2018bourgan,
  author = {Chang Xiao and Peilin Zhong and Changxi Zheng},
  title = {{BourGAN: Generative Networks with Metric Embeddings}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2018},
  url = {https://openreview.net/forum?id=rJEMew-uWH},
}
Figure