Diversity-Sensitive Conditional Generative Adversarial Networks

Dingdong Yang, Seunghoon Hong, Yunseok Jang, Tianchen Zhao, Honglak Lee

Introduction

Conditional Generative Adversarial Networks (cGANs) are powerful tools designed to map an input (such as a semantic label map or a sketch) to a realistic output (like a photograph). In many of these tasks, the mapping is inherently multi-modal; for instance, there are many valid ways to color a sketch or complete an occluded image. Ideally, a cGAN should use a random latent code ($z$) to explore these diverse possibilities.

However, in practice, cGANs frequently suffer from mode collapse. The generator often learns to ignore the random noise $z$ entirely, settling on a single "safe" output that satisfies the discriminator. This results in a deterministic model that fails to produce diverse results. While existing solutions like BicycleGAN address this, they often require complex architectural changes, such as additional encoders, making them difficult to adapt to new tasks.

Methodology: The Diversity-Sensitive GAN (DSGAN)

Researchers from the University of Michigan and Google Brain have introduced a "plug-and-play" regularization term that effectively mitigates mode collapse. Their approach, called Diversity-Sensitive GAN (DSGAN), adds a simple regularization term to the generator's loss function without requiring structural changes to the network.

The core idea is to maximize the ratio of the distance between two generated images to the distance between their corresponding latent codes:

$$L_z(G) = \mathbb{E}_{z_1, z_2} \left[ \min \left( \frac{\|G(x, z_1) - G(x, z_2)\|}{\|z_1 - z_2\|}, \tau \right) \right]$$

How it Works

  1. Distance Ratio: The term encourages the generator to ensure that different random seeds ($z_1$ and $z_2$) result in perceptually different images.
  2. Numerical Stability: A threshold parameter $\tau$ prevents the gradient from exploding when latent codes are very close.
  3. Feature Matching Variant: For tasks like face inpainting, where pixel-wise distance is a poor proxy for semantic change, the authors introduced DSGAN-FM, which measures distance in the feature space of the discriminator.

This method is model-agnostic, allowing it to be integrated into existing architectures like pix2pix or pix2pixHD with minimal effort.

Experimental Setup

The authors evaluated DSGAN across three distinct domains to demonstrate its versatility:

  • Image-to-Image Translation: Utilizing Cityscapes (labels $\to$ photo), Maps (satellite $\to$ map), and Edges $\to$ Shoes/Handbags.
  • Image Inpainting: Using the CelebA dataset with center-masked face images.
  • Video Prediction: Predicting future frames in the BAIR robot pushing and KTH human action datasets.

Key Insights and Results

Quantitative and qualitative analysis of the diversity-quality trade-off. As the regularization weight λ increases, diversity (LPIPS) improves while quality (FID) reaches an optimum before degrading.

The Quality vs. Diversity Trade-off

A critical component of DSGAN is the hyperparameter $\lambda$, which controls the strength of the diversity regularization. As shown in the analysis, increasing $\lambda$ leads to higher diversity (measured by LPIPS). Interestingly, the image quality (measured by FID) often improves initially as the model avoids mode collapse, but may eventually degrade if $\lambda$ is set too high, forcing the model away from the true data distribution.

Performance Benchmarks

DSGAN consistently outperformed both standard cGANs and the more complex BicycleGAN:

  • Diversity: On the label → image task, DSGAN achieved an LPIPS score of 0.18, significantly higher than BicycleGAN (0.15) and standard cGAN (0.01).
  • Quality: It maintained superior image quality, with a Frechet Inception Distance (FID) of 57.20, outperforming BicycleGAN's 62.95 (lower is better).
  • Latent Space Smoothness: Interpolation experiments revealed that DSGAN learns a smooth manifold. Transitioning between latent codes results in gradual, realistic changes in the output, such as a face slowly changing expression.

Conclusion and Future Work

DSGAN provides a simple yet effective remedy for mode collapse in conditional generative tasks. By providing a lower bound on the generator's gradient norm with respect to the latent code, it prevents the vanishing gradient issue that typically leads to deterministic behavior.

While highly effective, future work could focus on automatically learning the optimal balance between diversity and quality ($\lambda$) and numerical stability ($\tau$), potentially allowing the model to match the true data distribution without manual hyperparameter tuning.

Summary updated

Abstract

We propose a simple yet highly effective method that addresses the mode-collapse problem in the Conditional Generative Adversarial Network (cGAN). Although conditional distributions are multi-modal (i.e., having many modes) in practice, most cGAN approaches tend to learn an overly simplified distribution where an input is always mapped to a single output regardless of variations in latent code. To address such issue, we propose to explicitly regularize the generator to produce diverse outputs depending on latent codes. The proposed regularization is simple, general, and can be easily integrated into most conditional GAN objectives. Additionally, explicit regularization on generator allows our method to control a balance between visual quality and diversity. We demonstrate the effectiveness of our method on three conditional generation tasks: image-to-image translation, image inpainting, and future video prediction. We show that simple addition of our regularization to existing models leads to surprisingly diverse generations, substantially outperforming the previous approaches for multi-modal conditional generation specifically designed in each individual task.
BibTeX
@inproceedings{yang2019diversity,
  author = {Dingdong Yang and Seunghoon Hong and Yunseok Jang and Tianchen Zhao and Honglak Lee},
  title = {{Diversity-Sensitive Conditional Generative Adversarial Networks}},
  booktitle = {International Conference on Learning Representations},
  year = {2019},
  url = {https://openreview.net/forum?id=aRLMZYVGg4},
}
Figure