Reviving Undersampling for Long-Tailed Learning

Hao Yu, Yingxiao Du, Jianxin Wu

TL;DR

This paper introduces Balanced Training and Merging (BTM), a simple yet effective plug-and-play strategy to improve the performance of under-represented classes in long-tailed learning. By reviving the concept of balanced undersampling and merging models trained on small, balanced subsets, BTM significantly boosts harmonic and geometric mean accuracy with minimal overhead.

What Problem Does This Paper Address?

Deep learning models excel on balanced datasets, but real-world data often exhibits a "long-tailed" distribution, where a few classes have many samples (head classes) and many classes have very few samples (tail classes). This imbalance severely degrades performance on tail classes, leading to unfair and unreliable models. While existing methods aim to improve overall accuracy, they often overlook the worst-performing tail classes. Traditional evaluation metrics like average accuracy can mask these deficiencies. The paper addresses the challenge of effectively improving the performance of these under-represented classes without sacrificing overall accuracy or introducing significant computational costs.

Methods

The core of the proposed method, Balanced Training and Merging (BTM), involves a few key steps:

  • Balanced Training: The authors revive the idea of balanced undersampling. Instead of using the entire imbalanced dataset, they create multiple small, balanced subsets, each containing an equal number of samples from all classes. These subsets are then used to fine-tune a pre-trained model.
  • Model Merging: After fine-tuning on several balanced subsets, the resulting models are merged by simply averaging their weights. This creates a single, robust model.
  • Plug-and-Play Integration: BTM is designed to be a lightweight, plug-and-play module that can be easily integrated into existing long-tailed learning frameworks, particularly two-stage decoupling methods. It can be inserted between the initial pre-training stage and the classifier fine-tuning stage.

Datasets

The paper evaluates BTM on three widely used long-tailed benchmark datasets:

  • Places-LT: Derived from the balanced Places dataset.
  • ImageNet-LT: Derived from the balanced ImageNet dataset.
  • iNaturalist2018: A naturally imbalanced dataset.

The long-tailed versions of Places and ImageNet were constructed by down-sampling the original training sets using a Pareto distribution.

Key Insights and Ideas

  • Reviving Undersampling: Contrary to the common belief that balanced undersampling leads to severe underfitting, the authors demonstrate that fine-tuning a pre-trained model on few-shot balanced subsets can surprisingly improve the harmonic and geometric mean accuracy of tail classes.
  • Harmonic Mean as a Better Metric: The paper argues that the harmonic mean is a more sensitive metric than average accuracy for evaluating performance on tail classes, as it penalizes low per-class accuracies more heavily.
  • Model Averaging for Robustness: Merging models trained on different balanced subsets through simple weight averaging leads to further improvements in harmonic and geometric mean, creating a more uniform per-class accuracy distribution.
  • Minimal Overhead: BTM is designed to be efficient, requiring only a small amount of training data and minimal additional computational overhead, with no increase in inference cost.

Key Results

  • BTM significantly improves the harmonic and geometric mean accuracy across multiple long-tailed datasets (Places-LT, ImageNet-LT, iNaturalist2018) when applied to existing methods like MiSLAS and GPaCo.
  • For example, on Places-LT, applying BTM to GPaCo boosted the harmonic mean from 10.93% to 29.36% while only slightly reducing overall accuracy.
  • The method consistently maintains or slightly improves overall accuracy while substantially boosting the performance of the worst-performing categories.
  • Ablation studies show that fine-tuning the whole model (backbone and classifier) on balanced subsets yields better results than fine-tuning only the backbone or classifier.

Limitations and Future Work

  • Minimum Recall: While BTM significantly improves harmonic and geometric mean, the minimum recall (accuracy of the absolute worst class) can still remain zero on very large datasets like ImageNet and iNaturalist2018.
  • Arithmetic Accuracy Trade-off: In some scenarios, BTM might slightly decrease arithmetic accuracy.
  • Future Directions: The authors suggest exploring methods to further enhance minimum recall and to achieve simultaneous improvement of both average accuracy and harmonic/geometric mean. They also propose investigating how to combine BTM with other advanced techniques.

Summary updated

Abstract

The training datasets used in long-tailed recognition are extremely unbalanced, resulting in significant variation in per-class accuracy across categories. Prior works mostly used average accuracy to evaluate their algorithms, which easily ignores those worst-performing categories. In this paper, we aim to enhance the accuracy of the worst-performing categories and utilize the harmonic mean and geometric mean to assess the model's performance. We revive the balanced undersampling idea to achieve this goal. In few-shot learning, balanced subsets are few-shot and will surely under-fit, hence it is not used in modern long-tailed learning. But, we find that it produces a more equitable distribution of accuracy across categories with much higher harmonic and geometric mean accuracy, and, but lower average accuracy. Moreover, we devise a straightforward model ensemble strategy, which does not result in any additional overhead and achieves improved harmonic and geometric mean while keeping the average accuracy almost intact when compared to state-of-the-art long-tailed learning methods. We validate the effectiveness of our approach on widely utilized benchmark datasets for long-tailed learning. Our code is at \href{https://github.com/yuhao318/BTM/}{https://github.com/yuhao318/BTM/}.
BibTeX
@misc{yu2024reviving,
  author = {Hao Yu and Yingxiao Du and Jianxin Wu},
  title = {{Reviving Undersampling for Long-Tailed Learning}},
  year = {2024},
  eprint = {2401.16811},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2401.16811},
}
Figure