Class Balancing GAN with a Classifier in the Loop

Harsh Rangwani, Konda Reddy Mopuri, R. Venkatesh Babu

TL;DR

This paper tackles the challenge of training Generative Adversarial Networks (GANs) on imbalanced datasets, a common real-world scenario where standard GANs struggle. They introduce a novel "class-balancing regularizer" that leverages a pre-trained classifier to guide the GAN towards generating a more uniform distribution of classes, significantly improving image generation quality on long-tailed datasets.

What Problem Does This Paper Address?

Generative Adversarial Networks (GANs) have become incredibly adept at creating realistic images that mimic complex data distributions. However, most research and development have focused on GANs trained on balanced datasets, where each class has an equal number of samples. In reality, many datasets are "long-tailed," meaning a few classes have many samples (head classes), while most classes have very few (tail classes). Standard GANs, when trained on such imbalanced data, tend to over-represent the head classes and under-represent or even completely ignore the tail classes, leading to poor generation quality and a lack of diversity across all classes. This paper aims to bridge this gap by developing a method that enables GANs to effectively learn from and generate samples for imbalanced datasets.

Methods

The core contribution of this paper is a novel Class Balancing Regularizer. This regularizer is designed to be added to the generator's loss function during GAN training. Here's how it works:

  • Pre-trained Classifier: The method relies on a pre-trained classifier that can predict the class of an image. This classifier is trained separately, ideally on the same (potentially imbalanced) dataset.
  • Effective Class Frequency: The regularizer models the "effective class frequency" of the samples generated by the GAN. This is inspired by the concept of "catastrophic forgetting" in neural networks, where a network trained on new data tends to forget older data. They use an exponential forgetting mechanism to track how often the GAN generates samples from each class over time.
  • Balancing Objective: The regularizer then penalizes the GAN for over-generating samples from majority classes and encourages it to focus on under-represented (tail) classes. This is achieved by minimizing a specific objective function that relates the generated class distribution to the estimated effective class frequencies.
  • Theoretical Foundation: The paper provides a theoretical analysis, including a proposition that bounds the probability of generating samples from a particular class, demonstrating how the regularizer encourages a more uniform distribution.

This approach differs from standard GANs by explicitly incorporating class distribution information during training, guided by an external classifier, rather than solely relying on the discriminator's real/fake signal.

Datasets

The authors evaluated their method on several datasets:

  • CIFAR-10: A standard image classification dataset. They created imbalanced versions of this dataset.
  • LSUN: A large-scale dataset for scene understanding. They used a subset with 5 balanced classes.
  • CIFAR-100: A dataset with 100 classes, used to test performance on datasets with a larger number of classes.
  • iNaturalist-2019: A large-scale, naturally long-tailed dataset of species images, which is particularly challenging for GANs.

Key Insights and Ideas

  • GANs struggle with imbalance: The paper clearly demonstrates that standard GANs, even conditional ones, perform poorly on long-tailed datasets, producing biased distributions and lower quality images.
  • A pre-trained classifier is a powerful guide: By using a pre-trained classifier, the GAN gains crucial information about class distribution that it wouldn't otherwise have, enabling it to learn more effectively from imbalanced data.
  • "Effective class frequency" captures GAN behavior: The concept of modeling class frequency with an exponential forgetting mechanism provides a theoretically grounded way to understand and regularize GAN generation patterns.
  • Balancing improves both quality and diversity: The proposed regularizer not only leads to more balanced class distributions but also significantly improves the Fréchet Inception Distance (FID), a key metric for image generation quality.
  • Robustness to classifier accuracy: The method shows remarkable robustness; even with a classifier that has moderate accuracy on tail classes, the GAN can still achieve good results.

Key Results

  • State-of-the-art FID on iNaturalist-2019: When applied to an unconditional GAN on the long-tailed iNaturalist-2019 dataset, their method improved the FID from 13.03 to 9.01.
  • Improved FID on CIFAR-10: For CIFAR-10 with an imbalance ratio of 100, their method achieved an FID of 30.48, outperforming baselines like SNDCGAN (32.53) and ACGAN (38.33).
  • Balanced Class Distributions: The proposed method consistently achieved lower KL Divergence scores (indicating more uniform class distributions) compared to other baselines on imbalanced datasets.
  • Competitive performance on CIFAR-100: On CIFAR-100, their method achieved balanced distributions similar to CGAN and competitive FID scores.
  • Qualitative improvements: Figures in the paper show visually more diverse and balanced image generation from their method compared to baselines, especially for tail classes.

Limitations and Future Work

  • Dependency on a pre-trained classifier: While the paper argues that a pre-trained classifier is not overly burdensome, its presence is a requirement. The performance of the GAN is still somewhat dependent on the quality of this classifier, especially for extreme imbalance ratios.
  • Mode Collapse in some baselines: The paper notes that some baselines, even when using a pre-trained classifier, can suffer from mode collapse, which their method helps to avoid.
  • Further improvements: The authors suggest that their method could be combined with other techniques like consistency regularization or latent optimization to further enhance image quality. They also mention that there is scope for deeper understanding of the optimization dynamics involved.

Summary updated

Abstract

Generative Adversarial Networks (GANs) have swiftly evolved to imitate increasingly complex image distributions. However, majority of the developments focus on performance of GANs on balanced datasets. We find that the existing GANs and their training regimes which work well on balanced datasets fail to be effective in case of imbalanced (i.e. long-tailed) datasets. In this work we introduce a novel theoretically motivated Class Balancing regularizer for training GANs. Our regularizer makes use of the knowledge from a pre-trained classifier to ensure balanced learning of all the classes in the dataset. This is achieved via modelling the effective class frequency based on the exponential forgetting observed in neural networks and encouraging the GAN to focus on underrepresented classes. We demonstrate the utility of our regularizer in learning representations for long-tailed distributions via achieving better performance than existing approaches over multiple datasets. Specifically, when applied to an unconditional GAN, it improves the FID from $13.03$ to $9.01$ on the long-tailed iNaturalist-$2019$ dataset.
BibTeX
@inproceedings{rangwani2021class,
  author = {Harsh Rangwani and Konda Reddy Mopuri and R. Venkatesh Babu},
  title = {{Class Balancing GAN with a Classifier in the Loop}},
  booktitle = {Conference on Uncertainty in Artificial Intelligence},
  year = {2021},
  url = {https://openreview.net/forum?id=LkIbXpgm0G},
}

Loading related content...

Figure