Diversity-Sensitive Conditional Generative Adversarial Networks

Dingdong Yang, Seunghoon Hong, Yunseok Jang, Tianchen Zhao, Honglak Lee

Overview

Conditional GANs (cGANs) often suffer from "mode collapse," a phenomenon where the model ignores random noise inputs and produces the exact same output for a given condition every time. This paper introduces DSGAN, a simple yet highly effective regularization term that forces the generator to produce diverse outputs proportional to the distance between latent codes. This approach effectively restores multi-modality to image and video generation tasks without requiring complex architectural changes.

The Problem: Mode Collapse in cGANs

In an ideal scenario, a Conditional GAN taking an input (like a sketch or a segmentation map) and a random latent code (noise) should generate many plausible variations of the output, such as different colors, textures, or styles. However, in practice, cGANs frequently fall into mode collapse. The generator learns that the easiest way to "fool" the discriminator is to ignore the random noise entirely and map each input to a single, deterministic "safe" output.

This is a major hurdle for tasks where the outcome is inherently ambiguous. For example, in video prediction, there are many possible ways a person might move their arm; a model that only predicts one "average" path fails to capture the true complexity of the data. Previous solutions, like BicycleGAN, attempted to fix this by adding extra encoder networks to "force" the model to remember the noise, but these architectures are often complex to train and task-specific.

The DSGAN Method

The authors propose a "Diversity-Sensitive" regularization term that is remarkably simple. Instead of complex architectural additions, they add a single objective to the generator's loss function:

$$L_z(G) = \mathbb{E}_{z_1, z_2} \left[ \min \left( \frac{\|G(x, z_1) - G(x, z_2)\|}{\|z_1 - z_2\|}, \tau \right) \right]$$

How it works: 1. Distance Ratio: The model picks two different random latent codes ($z_1$ and $z_2$) for the same input $x$. 2. Proportional Change: It calculates the distance between the two resulting generated images. The loss encourages the generator to make the distance between the images at least proportional to the distance between the latent codes. 3. The Threshold ($\tau$): A cap ($\tau$) is used to prevent the generator from creating "exploding" differences that look unrealistic, ensuring numerical stability.

This approach is "plug-and-play"—it can be added to almost any existing cGAN framework (like pix2pix or pix2pixHD) by simply adding this term to the loss function.

Key Insights and Ideas

  • Gradient Connection: The authors show mathematically that maximizing the distance between outputs is equivalent to preventing the generator's gradient (with respect to the latent code) from vanishing. By keeping the gradient "alive," the model is forced to remain sensitive to the noise.
  • Semantic vs. Pixel Diversity: In image inpainting tasks, the authors found that regularizing based on discriminator features (DSGAN-FM) rather than raw pixels led to more "meaningful" diversity. Instead of just changing skin tone or lighting, the model started varying high-level features like facial expressions, age, and identity.
  • Scalability to Latent Space: Unlike BicycleGAN, which struggles as the dimensionality of the latent code $z$ increases, DSGAN remains stable and consistent across various latent code sizes.
  • The Quality-Diversity Trade-off: There is a natural "Pareto frontier" in generative modeling. By tuning the weight ($\lambda$) of the diversity term, users can choose whether they want extremely high-quality (but similar) images or highly diverse (but potentially noisier) results.
Quality-Diversity Trade-off

Key Results

The researchers evaluated DSGAN across several diverse benchmarks, including image-to-image translation (label→image, edge→photo, map→image), high-resolution synthesis, image inpainting, and video prediction.

  • Superior Diversity: DSGAN consistently achieved higher LPIPS scores (a metric for perceptual diversity) than standard cGANs and BicycleGAN across all image translation tasks.
Diverse outputs across multiple image translation tasks
  • Better Realism: On the map→image and label→image tasks, DSGAN achieved lower FID scores (indicating higher image quality/realism) than BicycleGAN.
  • Video Breakthrough: In video prediction on the KTH dataset, DSGAN significantly outperformed SAVP (a state-of-the-art multi-modal video model). While SAVP often collapsed to a single motion, DSGAN successfully generated diverse futures (e.g., a person waving vs. clapping) from the same starting frames.
  • High-Res Success: DSGAN was successfully integrated into pix2pixHD, producing diverse $1024 \times 512$ images, a feat that is architecturally difficult for models requiring bi-directional encoders.

Limitations and Future Work

  • Hyperparameter Sensitivity: The balance between diversity and quality depends heavily on the weight $\lambda$ and the threshold $\tau$. Currently, these must be tuned manually for different tasks.
  • Mode Balancing: While DSGAN prevents mode collapse, it doesn't explicitly guarantee that the proportions of the generated modes match the real-world distribution. Automatically learning $\lambda$ to match the target distribution is a promising future direction.
  • Computational Overhead: While conceptually simple, the method requires generating at least two images per training step to calculate the distance ratio, which increases training time compared to a standard cGAN.

Summary updated

Abstract

We propose a simple yet highly effective method that addresses the mode-collapse problem in the Conditional Generative Adversarial Network (cGAN). Although conditional distributions are multi-modal (i.e., having many modes) in practice, most cGAN approaches tend to learn an overly simplified distribution where an input is always mapped to a single output regardless of variations in latent code. To address such issue, we propose to explicitly regularize the generator to produce diverse outputs depending on latent codes. The proposed regularization is simple, general, and can be easily integrated into most conditional GAN objectives. Additionally, explicit regularization on generator allows our method to control a balance between visual quality and diversity. We demonstrate the effectiveness of our method on three conditional generation tasks: image-to-image translation, image inpainting, and future video prediction. We show that simple addition of our regularization to existing models leads to surprisingly diverse generations, substantially outperforming the previous approaches for multi-modal conditional generation specifically designed in each individual task.
BibTeX
@inproceedings{yang2019diversity,
  author = {Dingdong Yang and Seunghoon Hong and Yunseok Jang and Tianchen Zhao and Honglak Lee},
  title = {{Diversity-Sensitive Conditional Generative Adversarial Networks}},
  booktitle = {International Conference on Learning Representations},
  year = {2019},
  url = {https://openreview.net/forum?id=aRLMZYVGg4},
}
Figure