Label Augmentation as Inter-class Data Augmentation for Conditional Image Synthesis with Imbalanced Data

Kai Katsumata, Duc Minh Vo, Hideki Nakayama

Label Augmentation for Conditional Image Synthesis with Imbalanced Data

Abstract

Training conditional Generative Adversarial Networks (cGANs) on imbalanced datasets often results in "mode collapse" for minority classes or the generation of images that lack semantic alignment with their labels. This paper introduces Softlabel-GAN, a novel framework that utilizes a pretrained classifier to assign "soft labels" to real images during training. By treating data augmentation as an inter-class process, minority classes can effectively leverage diverse features from semantically similar, well-represented classes. Experimental results across multiple datasets demonstrate that Softlabel-GAN significantly improves image quality and diversity compared to existing state-of-the-art augmentation techniques.

The Challenge of Imbalanced Data

Conditional Image Synthesis has achieved remarkable results when trained on large, balanced datasets. However, real-world data is frequently imbalanced, characterized by a "long-tail" distribution where a few majority classes dominate and many minority classes have very few samples.

Standard data augmentation (DA) techniques often fail in these scenarios in two specific ways:

  1. Low Diversity: Label-preserving DA (e.g., flipping or rotating) simply reuses the same few samples, leading the generator to memorize and repeat the same images (mode collapse).
  2. Low Affinity: Non-label-preserving DA (e.g., Mixup or random label smoothing) might assign a "cat" image a "dog" label, confusing the model with features that do not belong to the target class.

Methods: Softlabel-GAN

The proposed Softlabel-GAN addresses these issues through Label Augmentation, which modifies labels based on image content rather than just augmenting the images themselves.

The Softlabel-GAN architecture incorporates a pretrained classifier (C) to provide soft labels for real images, which are then used by the discriminator to compute the adversarial loss.

The architecture consists of several key components:

  1. Pretrained Classifier: A standard classifier is trained on the imbalanced dataset to capture inter-class relationships.
  2. Soft Labels for Real Images: During GAN training, the discriminator receives a probability vector (soft label) from the pretrained classifier for each real image. This vector reflects the image's similarity to various classes (e.g., a rare "Persian Cat" might share features with a common "Ragdoll Cat").
  3. Discriminator Training: The discriminator learns to associate these content-aware soft labels with real images, forcing it to recognize shared features across similar classes. This prevents the discriminator from easily "winning" by just memorizing the few available real samples in a minority class.
  4. Generator Training: While the generator still receives standard one-hot labels, it benefits from the discriminator's richer understanding of class boundaries, allowing it to produce more diverse images for minority classes by "borrowing" statistical strength from their common neighbors.

Key Results and Insights

Softlabel-GAN was evaluated on several challenging benchmarks, including AnimeFace, Oxford-102 Flowers, and imbalanced versions of CIFAR and Tiny ImageNet. The method consistently outperformed baselines like DiffAug-GAN and Smooth-GAN.

Qualitative comparison on the AnimeFace dataset. Red boxes highlight mode collapse in DiffAug and Smooth-GAN, whereas Softlabel-GAN (Ours) maintains high diversity and fidelity.
  • Image Quality (FID): On the AnimeFace dataset, Softlabel-GAN achieved an FID of 19.14, a significant improvement over DiffAug-GAN's 25.09. Lower FID indicates higher image quality and better distribution matching.
  • Diversity (LPIPS): Higher LPIPS scores confirmed that the model generates varied images rather than repeating training samples.
  • Robustness to Scarcity: The performance remained stable even when minority class samples were extremely scarce (down to 65 samples), where other methods saw their FID scores skyrocket.

Conclusion

Softlabel-GAN provides a robust solution for conditional image synthesis in data-scarce and imbalanced regimes. By using content-aware soft labels, it bridges the gap between classes and prevents the discriminator from over-memorizing minority samples. While dependent on a pretrained classifier, the approach offers a significant step forward in making generative models more applicable to real-world, long-tailed data distributions. Future work may explore applying these principles to other architectures, such as Diffusion Models.

Summary updated

Abstract

Conditional image synthesis performs admirably when trained on well-constructed and balanced datasets. However, in practice, training datasets frequently contain minorities (i.e., a class with a few samples), known as imbalanced data, which causes difficulties in learning generative models. To address conditional image synthesis with imbalanced data, we analyze a diversity issue of label-preserving data augmentation and an affinity issue of non-label-preserving data augmentation. From this observation, we present label augmentation, which works as inter-class data augmentation that effectively augments data by predicting a new label for a given image using the prediction of a pretrained image classification model (i.e., probabilities for each class). We incorporate our label augmentation into the discriminator of a seminal conditional generative adversarial network (GAN) model, proposing Softlabel-GAN. Using class probabilities extracts class-invariant and shared features between similar classes, achieving data augmentation with high affinity and diversity. Our experiments on imbalanced datasets show that Softlabel-GAN produces images with high quality and diversity while being hardly affected by the number of samples in each class. Code: https://github.com/raven38/softlabel-gan.
BibTeX
@inproceedings{katsumata2024label,
  author = {Kai Katsumata and Duc Minh Vo and Hideki Nakayama},
  title = {{Label Augmentation as Inter-class Data Augmentation for Conditional Image Synthesis with Imbalanced Data}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2024},
  doi = {10.1109/wacv57701.2024.00487},
}
Figure