RSG: A Simple but Effective Module for Learning Imbalanced Datasets

Jianfeng Wang, Thomas Lukasiewicz, Xiaolin Hu, Jianfei Cai, Zhenghua Xu

Abstract

Deep learning models often struggle with long-tailed data distributions where a few "head" classes dominate and many "tail" classes have sparse representation. We present the Rare-class Sample Generator (RSG), a plug-and-play module that addresses this imbalance by transferring intra-class variations—such as lighting, pose, and background—from frequent classes to rare ones. By "hallucinating" realistic new samples in the feature space during training, RSG enables models to learn more robust decision boundaries for infrequent categories. RSG is computationally efficient, discarded after training, and achieves state-of-the-art results on major imbalanced benchmarks including iNaturalist 2018 and ImageNet-LT.

Introduction

In real-world applications, data distributions are rarely uniform. While curated datasets like ImageNet provide balanced class counts, natural data typically follows a "long-tail" distribution. Standard deep learning models trained on such data become biased toward frequent classes, leading to poor generalization on rare categories.

Common strategies to mitigate this include oversampling rare classes (which risks overfitting), reweighting loss functions (which can destabilize training), or using generative models like GANs (which are complex and often fail to capture realistic intra-class variations). RSG offers a simpler, more effective alternative by leveraging the rich variation information already present in the frequent classes of the dataset.

Overview of the RSG module integrated into a CNN pipeline.

The RSG Module

The RSG module is designed to be integrated into any standard Convolutional Neural Network (CNN) architecture, such as ResNet or DenseNet. Its primary function is to extract "feature displacements"—vectors representing class-irrelevant variations—from frequent classes and apply them to rare classes to generate synthetic training samples.

Core Components

The module consists of three primary sub-modules:

  1. Center Estimation Module: This component estimates the centroid of each class within the feature space, providing a reference point for calculating variations.
  2. Vector Transformation Module: It computes the displacement between a specific sample's features and its class center. This displacement captures variations like camera angle or illumination that are independent of the class identity.
  3. Contrastive Module: To ensure that the transferred variations do not include class-defining features (e.g., transferring a bird's wing to a car), the Contrastive Module acts as a filter, ensuring the extracted displacements remain class-agnostic.
Detailed architecture of the RSG module showing its three core components.

Training Workflow

During the training phase, RSG identifies a "displacement" from a frequent-class sample and adds it to a real rare-class sample. This creates a "hallucinated" feature map that represents the rare class under a new variation. To further improve the classifier, the authors employ a Maximized Vector (MV) loss. This loss function encourages generated samples to lie near decision boundaries, forcing the network to learn more precise separators between classes. Notably, RSG is only active during training; it is removed for inference, ensuring no added latency during deployment.

Experimental Results

The effectiveness of RSG was validated across several large-scale imbalanced datasets:

  • ImageNet-LT & Places-LT: On ImageNet-LT, RSG reached 51.8% accuracy, with significant improvements in "few-shot" classes. On Places-LT, it achieved 39.3% top-1 accuracy, surpassing previous benchmarks like OLTR.
  • iNaturalist 2018: In this naturally imbalanced dataset of over 8,000 species, RSG reduced the error rate to 29.74%, outperforming the BBN baseline.
  • CIFAR-10/100: On synthetic long-tailed versions of CIFAR, RSG consistently improved performance over strong baselines like LDAM-DRW.

Conclusion

RSG provides a robust and flexible solution for the long-tail problem in machine learning. By intelligently transferring variations across classes, it enriches the feature space of rare categories without the need for complex generative architectures or additional data. Its "plug-and-play" nature and zero-cost inference make it a highly practical tool for real-world imbalanced learning tasks.

Summary updated

Abstract

Imbalanced datasets widely exist in practice and are a great challenge for training deep neural models with a good generalization on infrequent classes. In this work, we propose a new rare-class sample generator (RSG) to solve this problem. RSG aims to generate some new samples for rare classes during training, and it has in particular the following advantages: (1) it is convenient to use and highly versatile, because it can be easily integrated into any kind of convolutional neural network, and it works well when combined with different loss functions, and (2) it is only used during the training phase, and therefore, no additional burden is imposed on deep neural networks during the testing phase. In extensive experimental evaluations, we verify the effectiveness of RSG. Furthermore, by leveraging RSG, we obtain competitive results on Imbalanced CIFAR and new state-of-the-art results on Places-LT, ImageNet-LT, and iNaturalist 2018. The source code is available at https://github.com/Jianf-Wang/RSG.
BibTeX
@inproceedings{wang2021rsg,
  author = {Jianfeng Wang and Thomas Lukasiewicz and Xiaolin Hu and Jianfei Cai and Zhenghua Xu},
  title = {{RSG: A Simple but Effective Module for Learning Imbalanced Datasets}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2021},
  doi = {10.1109/cvpr46437.2021.00378},
}
Figure