Learning Generative Adversarial Networks from Multiple Data Sources

Trung Le, Quan Hoang, Hung Vu, Tu Dinh Nguyen, Hung Bui, Dinh Q. PhungIJCAI 2019 · OpenReview

Abstract

Generative Adversarial Networks (GANs) are a powerful class of deep generative models. In this paper, we extend GAN to the problem of generating data that are not only close to a primary data source but also required to be different from auxiliary data sources. For this problem, we enrich both GANs' formulations and applications by introducing pushing forces that thrust generated samples away from given auxiliary data sources. We term our method Push-and-Pull GAN (P2GAN). We conduct extensive experiments to demonstrate the merit of P2GAN in two applications: generating data with constraints and addressing the mode collapsing problem. We use CIFAR-10, STL-10, and ImageNet datasets and compute Fréchet Inception Distance to evaluate P2GAN's effectiveness in addressing the mode collapsing problem. The results show that P2GAN outperforms the state-of-the-art baselines. For the problem of generating data with constraints, we show that P2GAN can successfully avoid generating specific features such as black hair.

Introduction

Standard Generative Adversarial Networks (GANs) typically learn from a single data source, where a generator attempts to match a target distribution by fooling a discriminator. However, this paradigm faces two significant challenges: mode collapse and single-source limitations. Mode collapse occurs when the generator produces a limited variety of samples, failing to capture the full diversity of the target distribution. Furthermore, standard GANs lack a mechanism to explicitly avoid certain features without complex, labeled conditional setups.

To address these issues, we introduce Push-and-Pull GAN (P2GAN). This framework allows models to learn from multiple data sources simultaneously by balancing "pulling" forces (attraction to the target) and "pushing" forces (repulsion from unwanted data).

Methods: The Push-and-Pull Formulation

The core innovation of P2GAN is the extension of the standard $f$-divergence objective to include auxiliary sources.

  • The Pull: A primary discriminator $S(x)$ provides a "pulling force," encouraging the generator $G$ to match the primary data distribution.
  • The Push: One or more auxiliary discriminators $\tilde{S}_i(x)$ provide "pushing forces," rewarding the generator for producing samples that are distinct from auxiliary distributions.

Addressing Mode Collapse

To solve mode collapse, we employ a sequential training strategy. After training an initial generator, its output is treated as an auxiliary source. Subsequent generators are then "pushed" away from these previously learned distributions, forcing them to explore and capture missing modes of the data.

P2GAN capturing all 8 modes of a synthetic 2D Gaussian mixture by epoch 8k.

Constrained Generation

P2GAN enables unsupervised constrained generation. For example, to generate images with a specific feature (e.g., blond hair) from a mixed dataset, an auxiliary set containing the unwanted feature (e.g., black hair) can be used to "push" the generator toward the desired attribute. Unlike conditional GANs, this does not require labeling the primary dataset.

Key Results and Discussion

P2GAN was evaluated on synthetic 2D data, CIFAR-10, STL-10, ImageNet, and CelebA.

Quantitative Performance and Stability

P2GAN achieved state-of-the-art Fréchet Inception Distance (FID) scores. On CIFAR-10, it reached an FID of 20.1, outperforming SN-GAN (21.7) and MGAN (20.9). On STL-10, it achieved 38.2, compared to SN-GAN's 40.1.

Crucially, P2GAN exhibits superior training stability. While standard GANs often become unstable after many epochs, P2GAN's performance remains steady and continues to improve over time.

FID score comparison showing P2GAN's superior stability and lower distance compared to a standard GAN.

Qualitative Success

In the CelebA experiments, P2GAN successfully generated blond-haired faces by being pushed away from a black-haired auxiliary set. This demonstrates the effectiveness of the pushing force in steering the generator toward specific semantic regions of the data space without explicit labels.

Conclusion

P2GAN enriches the GAN framework by introducing pushing forces, providing a robust solution to mode collapse and a flexible method for constrained generation. While the pushing force hyperparameter requires tuning and sequential training adds computational cost, the gains in stability and diversity are significant. Future work will focus on more efficient single-generator architectures and applications in anomaly detection.

Summary updated

Abstract

Generative Adversarial Networks (GANs) are a powerful class of deep generative models. In this paper, we extend GAN to the problem of generating data that are not only close to a primary data source but also required to be different from auxiliary data sources. For this problem, we enrich both GANs' formulations and applications by introducing pushing forces that thrust generated samples away from given auxiliary data sources. We term our method Push-and-Pull GAN (P2GAN). We conduct extensive experiments to demonstrate the merit of P2GAN in two applications: generating data with constraints and addressing the mode collapsing problem. We use CIFAR-10, STL-10, and ImageNet datasets and compute Fréchet Inception Distance to evaluate P2GAN's effectiveness in addressing the mode collapsing problem. The results show that P2GAN outperforms the state-of-the-art baselines. For the problem of generating data with constraints, we show that P2GAN can successfully avoid generating specific features such as black hair.
BibTeX
@inproceedings{le2019learning,
  author = {Trung Le and Quan Hoang and Hung Vu and Tu Dinh Nguyen and Hung Bui and Dinh Q. Phung},
  title = {{Learning Generative Adversarial Networks from Multiple Data Sources}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2019},
  url = {https://openreview.net/forum?id=x36mZS2TwD},
}
Figure