Introduction
Conditional Generative Adversarial Networks (cGANs) are powerful tools designed to map an input (such as a semantic label map or a sketch) to a realistic output (like a photograph). In many of these tasks, the mapping is inherently multi-modal; for instance, there are many valid ways to color a sketch or complete an occluded image. Ideally, a cGAN should use a random latent code ($z$) to explore these diverse possibilities.
However, in practice, cGANs frequently suffer from mode collapse. The generator often learns to ignore the random noise $z$ entirely, settling on a single "safe" output that satisfies the discriminator. This results in a deterministic model that fails to produce diverse results. While existing solutions like BicycleGAN address this, they often require complex architectural changes, such as additional encoders, making them difficult to adapt to new tasks.
Methodology: The Diversity-Sensitive GAN (DSGAN)
Researchers from the University of Michigan and Google Brain have introduced a "plug-and-play" regularization term that effectively mitigates mode collapse. Their approach, called Diversity-Sensitive GAN (DSGAN), adds a simple regularization term to the generator's loss function without requiring structural changes to the network.
The core idea is to maximize the ratio of the distance between two generated images to the distance between their corresponding latent codes:
$$L_z(G) = \mathbb{E}_{z_1, z_2} \left[ \min \left( \frac{\|G(x, z_1) - G(x, z_2)\|}{\|z_1 - z_2\|}, \tau \right) \right]$$
How it Works
- Distance Ratio: The term encourages the generator to ensure that different random seeds ($z_1$ and $z_2$) result in perceptually different images.
- Numerical Stability: A threshold parameter $\tau$ prevents the gradient from exploding when latent codes are very close.
- Feature Matching Variant: For tasks like face inpainting, where pixel-wise distance is a poor proxy for semantic change, the authors introduced DSGAN-FM, which measures distance in the feature space of the discriminator.
This method is model-agnostic, allowing it to be integrated into existing architectures like pix2pix or pix2pixHD with minimal effort.
Experimental Setup
The authors evaluated DSGAN across three distinct domains to demonstrate its versatility:
- Image-to-Image Translation: Utilizing Cityscapes (labels $\to$ photo), Maps (satellite $\to$ map), and Edges $\to$ Shoes/Handbags.
- Image Inpainting: Using the CelebA dataset with center-masked face images.
- Video Prediction: Predicting future frames in the BAIR robot pushing and KTH human action datasets.
Key Insights and Results
The Quality vs. Diversity Trade-off
A critical component of DSGAN is the hyperparameter $\lambda$, which controls the strength of the diversity regularization. As shown in the analysis, increasing $\lambda$ leads to higher diversity (measured by LPIPS). Interestingly, the image quality (measured by FID) often improves initially as the model avoids mode collapse, but may eventually degrade if $\lambda$ is set too high, forcing the model away from the true data distribution.
Performance Benchmarks
DSGAN consistently outperformed both standard cGANs and the more complex BicycleGAN:
- Diversity: On the
label → imagetask, DSGAN achieved an LPIPS score of 0.18, significantly higher than BicycleGAN (0.15) and standard cGAN (0.01). - Quality: It maintained superior image quality, with a Frechet Inception Distance (FID) of 57.20, outperforming BicycleGAN's 62.95 (lower is better).
- Latent Space Smoothness: Interpolation experiments revealed that DSGAN learns a smooth manifold. Transitioning between latent codes results in gradual, realistic changes in the output, such as a face slowly changing expression.
Conclusion and Future Work
DSGAN provides a simple yet effective remedy for mode collapse in conditional generative tasks. By providing a lower bound on the generator's gradient norm with respect to the latent code, it prevents the vanishing gradient issue that typically leads to deterministic behavior.
While highly effective, future work could focus on automatically learning the optimal balance between diversity and quality ($\lambda$) and numerical stability ($\tau$), potentially allowing the model to match the true data distribution without manual hyperparameter tuning.
Summary updated