Mutual Learning for Long-Tailed Recognition

Changhwa Park, Junho Yim, Eunji JunWACV 2023

TL;DR

This paper tackles the challenge of recognizing objects in real-world datasets where some categories have vastly more examples than others (long-tailed distributions). They propose a "mutual learning" framework where multiple neural networks collaborate to learn better, more generalizable representations, leading to state-of-the-art performance on several long-tailed recognition benchmarks.

What Problem Does This Paper Address?

Deep neural networks excel on balanced datasets, but real-world data is often "long-tailed." This means a few categories (head classes) have abundant data, while many others (tail classes) have very few examples. Standard training on such data leads to models that perform well on head classes but poorly on tail classes. While previous work focused on designing better classifiers for imbalanced data, this paper argues that the quality of the learned representations (features) is a bottleneck. Poor representations, especially for tail classes which tend to overfit, limit even the best classifiers. The core problem is learning robust and generalizable features from imbalanced data.

Methods

The paper introduces a mutual learning framework for representation learning in long-tailed settings. The key idea is to train multiple neural network "experts" collaboratively.

  • Shared Backbone: All experts share an initial feature extractor (backbone) to reduce computational cost.
  • Mutual Learning Loss: Instead of just training each expert independently, they introduce a "mutual learning loss." This loss encourages the experts to learn similar representations by penalizing the divergence between their output probability distributions. This is inspired by deep mutual learning, where students teach each other. The intuition is that by exchanging information, the experts can help each other avoid falling into local minima, especially for tail classes where data is scarce.
  • Classification Loss: Each expert is also trained with a standard classification loss (cross-entropy) on its own predictions.
  • PC Softmax: For inference, they use a Post-Compensated Softmax (PC Softmax) classifier. This technique adjusts the model's output to better match the test dataset's label distribution, which is often different from the training distribution in long-tailed scenarios.

Datasets

The proposed method was evaluated on the following long-tailed benchmark datasets:

  • ImageNet-LT: A long-tailed version of ImageNet with 1000 classes and a power-law distribution of samples.
  • CIFAR100-LT: A long-tailed version of CIFAR-100, tested with various imbalance ratios (10, 50, 100).
  • iNaturalist 2018: A large-scale dataset for species classification with a high imbalance ratio.

Key Insights and Ideas

  • Representation Quality is Paramount: The paper strongly emphasizes that improving the quality of learned features is more critical than solely focusing on classifier design for long-tailed recognition.
  • Mutual Learning for Generalization: Collaborative learning among multiple experts, guided by a mutual learning loss, effectively prevents overfitting to tail classes and leads to more generalizable representations.
  • Sampling Strategy Matters (Even in Collaboration): The choice of sampling strategy during training is crucial. Instance-balanced sampling (sampling one image per class) consistently yields better representations than class-balanced sampling (sampling classes uniformly) even within the mutual learning framework.
  • PC Softmax Complements Feature Learning: While mutual learning improves feature quality, PC Softmax acts as an effective post-processing step to further boost performance by aligning model outputs with the test distribution.

Key Results

  • State-of-the-Art Performance: The proposed mutual learning framework with PC Softmax achieved state-of-the-art results on CIFAR100-LT, ImageNet-LT, and iNaturalist 2018.
  • Improved Feature Accuracy: Mutual learning improved feature accuracy (a measure of representation quality) by 3.1% compared to independent learning on CIFAR100-LT.
  • Significant Gains on Benchmarks:
    • ImageNet-LT: Outperformed RIDE by 2.8% and ACE by 2.9%.
  • CIFAR100-LT: Achieved new state-of-the-art results across all imbalance ratios (10, 50, 100), with improvements up to 2.0% over existing methods.
  • iNaturalist 2018: Showed a 2.0% performance improvement over ACE.
  • Sampling Strategy Impact: Instance-balanced sampling alone or in combination with class-balanced sampling (I+I or I+C) outperformed class-balanced sampling alone (C+C) in terms of feature accuracy.

Limitations and Future Work

  • Computational Cost: While the paper uses a shared backbone to mitigate this, training multiple experts inherently increases computational cost compared to a single network. The paper notes that their three-expert model is comparable to baseline models in terms of GFlops.
  • Adaptability of PC Softmax: The paper notes that while PC Softmax is effective with their method, applying it to other methods that already use balanced classifiers can sometimes lead to performance degradation (Table 8), suggesting it's best suited for methods that don't inherently balance predictions.
  • Future Work: The paper suggests that the mutual learning framework is simple and easily adaptable to other cutting-edge approaches, implying potential for integration with other long-tailed recognition techniques. The exploration of larger numbers of experts also suggests potential for further gains.

Summary updated

BibTeX
@inproceedings{park2023mutual,
  author = {Changhwa Park and Junho Yim and Eunji Jun},
  title = {{Mutual Learning for Long-Tailed Recognition}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2023},
}
Figure