Bag of Tricks for Long-Tailed Visual Recognition with Deep Convolutional Neural Networks

Yongshun Zhang, Xiu-Shen Wei, Boyan Zhou, Jianxin Wu

Abstract

Real-world visual data often follows a long-tailed distribution, where a few "head" classes dominate and many "tail" classes have sparse samples. Standard Convolutional Neural Networks (CNNs) are inherently biased toward majority classes, leading to poor performance on the tail. This paper presents a systematic empirical study of simple training refinements—re-sampling, re-weighting, and mixup—to identify which combinations work synergistically. We propose a novel Class Activation Map (CAM)-based data augmentation for tail classes. By combining these refinements into a "bag of tricks," we achieve state-of-the-art performance on major benchmarks like iNaturalist 2018 and ImageNet-LT without the complexity of meta-learning.

Introduction

In practical applications, datasets are rarely balanced. While standard benchmarks like ImageNet-2012 are curated for class parity, natural distributions are typically long-tailed. This imbalance causes CNNs to overfit to head classes while failing to generalize to rare ones. While recent research has proposed complex architectural or algorithmic solutions, many practitioners rely on "tricks" like over-sampling or focal loss. However, these tricks are often applied without a clear understanding of their interactions. This work aims to provide a scientific guideline for these refinements and introduces a targeted augmentation strategy to bolster tail-class representation.

Methodology: Decoupling and Augmentation

The authors categorize existing refinements into four families: re-weighting, re-sampling, mixup training, and two-stage training. A key insight is that re-balancing methods often conflict with the model's ability to learn high-quality general features.

Two-Stage Training

The most effective strategy is decoupling the learning process. In Stage 1, the model learns feature representations using the original imbalanced data (often with Mixup for regularization). In Stage 2, the features are frozen, and only the classifier is tuned using balancing tricks like Deferred Re-balancing (DRS).

CAM-Based Data Augmentation

To further improve tail-class performance, the authors introduce a novel augmentation technique. Using Class Activation Maps (CAM), the model identifies the most discriminative "foreground" pixels for a tail-class image. Transformations such as scaling, rotating, and flipping are applied specifically to this foreground while the background remains constant. This creates informative "jittered" samples that help the model learn robust features for rare classes without introducing excessive noise.

Overview of the proposed pipeline featuring CAM-based data augmentation for tail classes.

Experimental Results

The "bag of tricks" was evaluated on four major benchmarks: CIFAR-10-LT, CIFAR-100-LT, iNaturalist 2018, and ImageNet-LT. The optimal configuration includes Input Mixup in Stage 1, followed by DRS with CAM-based sampling and Cost-Sensitive Cross-Entropy in Stage 2.

As shown in the table below, the proposed approach consistently outperforms complex state-of-the-art methods like BBN, LDAM, and various meta-learning frameworks across all tested imbalance factors.

Comparison of the proposed 'bag of tricks' against state-of-the-art methods across major long-tailed benchmarks.

On the iNaturalist 2018 dataset, the error rate was reduced from a baseline of 39.89% to 29.13%. Similarly, on ImageNet-LT, the error rate dropped from 65.99% to 56.87%. These improvements are achieved without adding extra computational overhead during inference.

Conclusion

This study demonstrates that a systematic application of simple training refinements can surpass the performance of complex, specialized models for long-tailed recognition. The key to success lies in decoupling feature learning from classifier balancing and employing targeted data augmentation for tail classes. Future work will explore the applicability of these tricks to other domains such as object detection and video recognition.

Summary updated

Abstract

In recent years, visual recognition on challenging long-tailed distributions, where classes often exhibit extremely imbalanced frequencies, has made great progress mostly based on various complex paradigms (e.g., meta learning). Apart from these complex methods, simple refinements on training procedures also make contributions. These refinements, also called tricks, are minor but effective, such as adjustments in the data distribution or loss functions. However, different tricks might conflict with each other. If users apply these long-tail related tricks inappropriately, it could cause worse recognition accuracy than expected. Unfortunately, there has not been a scientific guideline of these tricks in the literature. In this paper, we first collect existing tricks in long-tailed visual recognition and then perform extensive and systematic experiments, in order to give a detailed experimental guideline and obtain an effective combination of these tricks. Furthermore, we also propose a novel data augmentation approach based on class activation maps for long-tailed recognition, which can be friendly combined with re-sampling methods and shows excellent results. By assembling these tricks scientifically, we can outperform state-of-the-art methods on four long-tailed benchmark datasets, including ImageNet-LT and iNaturalist 2018. Our code is open-source and available at https://github.com/zhangyongshun/BagofTricks-LT.
BibTeX
@inproceedings{zhang2021bag,
  author = {Yongshun Zhang and Xiu-Shen Wei and Boyan Zhou and Jianxin Wu},
  title = {{Bag of Tricks for Long-Tailed Visual Recognition with Deep Convolutional Neural Networks}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2021},
  doi = {10.1609/aaai.v35i4.16458},
}
Figure