TL;DR
This paper introduces MsConD, a novel multi-scale contrastive learning approach to improve GANs' ability to generate complex scene images. By enhancing the discriminator's understanding of local semantic structures across multiple scales, MsConD significantly boosts the realism and quality of generated scenes, overcoming limitations of existing methods that focus on global representations.
What Problem Does This Paper Address?
Generative Adversarial Networks (GANs) have achieved remarkable success in generating realistic single object images. However, they struggle with complex scene generation, which involves multiple objects with diverse arrangements, sizes, and spatial relationships. The core challenge lies in the discriminator's difficulty in learning the intricate, multi-layered differences between real and fake scenes, leading to unsatisfactory synthesis quality, limited distribution coverage, and messy layouts. Existing GAN improvements often focus on global image representations, which are insufficient for capturing the fine-grained local details crucial for realistic scene generation.
Methods
The paper proposes a Multi-scale Contrastive Discriminator (MsConD) that enhances GANs through two primary mechanisms:
- Multi-scale Discriminator Architecture: The discriminator is redesigned with multi-level branches, each processing feature maps at different scales. This allows the discriminator to extract local feedback from various resolutions of the input image. Each branch includes residual blocks, a classification head for real/fake prediction, and a projection head for generating auxiliary representations. This multi-scale feedback is then used to compute a per-pixel adversarial loss.
- Multi-scale Contrastive Learning: An auxiliary self-supervised pretext task, instance discrimination, is introduced. This task is applied at each scale level to enrich the region-level representations used by the discriminator. For a given image and its augmented version, features from corresponding local regions (identified by a distance threshold) are treated as positive pairs, while features from other images in the batch serve as negative pairs. This pixel-level contrastive loss is jointly optimized with the adversarial loss across all scales.
Datasets
The authors evaluate their method on three challenging scene image datasets:
- Cityscapes: Contains 25k street scene images.
- LSUN (Livingroom and Kitchen subsets): Large collections of indoor scenes, specifically chosen for their complex data distributions. The Livingroom dataset contains 1.3 million images, and the Kitchen dataset contains 2.2 million images.
Key Insights and Ideas
- Local Feedback is Crucial for Complex Scenes: The paper strongly advocates that for complex scenes, relying solely on global discriminator feedback is insufficient. Multi-scale local feedback is essential for capturing the intricate details and relationships between objects.
- Self-Supervised Learning for Discriminator Enhancement: Integrating a self-supervised task (contrastive learning) directly into the discriminator's training process is a powerful way to improve its discriminative capabilities, especially for nuanced local features.
- The Importance of Spatial Consistency in Contrastive Learning: For scene generation, simply treating random augmentations as positive pairs in contrastive learning is not enough. Identifying positive features based on spatial proximity (using a distance threshold) is critical to ensure that semantically related local regions are learned together.
- Multi-scale Approach Synergizes Adversarial and Contrastive Learning: The combination of multi-scale adversarial feedback and multi-scale contrastive learning creates a synergistic effect, leading to significantly better generation quality than either component alone.
Key Results
- Scene-level Metrics: MsConD outperforms state-of-the-art baselines on Cityscapes, Livingroom, and Kitchen datasets in terms of FID (Frechet Inception Distance) and KID (Kernel Inception Distance). For example, on Cityscapes, MsConD achieves an FID of 2.63, a significant improvement over StyleGAN2 (8.04) and InsGen (4.21). The method also demonstrates improved recall, indicating better synthesis of diverse scenes.
- Object-level Metrics: MsConD shows superior performance in generating individual object categories within scenes, as measured by FID and KID scores on detected object crops. This validates its ability to produce more realistic and well-formed local objects.
- Qualitative Results: Visual comparisons (Figure 3) show that MsConD generates more realistic scene components, better arranged layouts, and fewer artifacts compared to baselines. Ablation studies (Figure 5) highlight that removing multi-scale adversarial loss leads to incomplete objects, while removing contrastive learning results in repetitive patterns.
Limitations and Future Work
- Distance Threshold Sensitivity: The paper notes that the performance of the contrastive learning component is sensitive to the distance threshold ($t$). An optimal threshold is required to balance capturing relevant local features without including irrelevant ones. Future work could explore adaptive or learned thresholds.
- Computational Cost: While not explicitly detailed as a limitation, multi-scale architectures and contrastive learning can increase computational complexity.
- Generalizability to Other Domains: The current work focuses on scene generation. Future work could explore extending this multi-scale contrastive approach to other complex data domains or tasks.
Summary updated