Nested Collaborative Learning for Long-Tailed Visual Recognition

Jun Li, Zichang Tan, Jun Wan, Zhen Lei, Guodong Guo

Abstract

Long-tailed visual recognition remains a significant challenge in computer vision due to the extreme imbalance between common "head" classes and rare "tail" classes. Standard deep learning models often exhibit high learning uncertainty and bias toward majority categories. This paper presents NCL++, a Nested Collaborative Learning framework designed to stabilize the learning process through inter-expert and intra-expert consistency. By combining balanced online distillation with nested feature learning, NCL++ significantly improves accuracy for rare categories while maintaining computational efficiency during inference. Experimental results across five major benchmarks, including iNaturalist 2018 and ImageNet-LT, demonstrate that NCL++ achieves state-of-the-art performance.

Introduction

In real-world scenarios, data distribution is rarely uniform. While curated datasets like ImageNet provide balanced class distributions, natural data typically follows a "long-tail" distribution. In these settings, a few head classes dominate the dataset, while hundreds of tail classes contain very few samples. Standard models trained on such data become biased, frequently misclassifying rare samples as common ones.

The authors of NCL++ identify learning uncertainty as a primary driver of this failure. They observed that identical models trained on imbalanced data often produce inconsistent predictions for the same image, a problem exacerbated by different data augmentations. NCL++ aims to mitigate this instability by forcing multiple "experts" and various views of the same image to reach a consensus through a structured collaborative framework.

Methodology

NCL++ is built upon three primary pillars that address uncertainty and bias through nested learning objectives.

1. Collaborative Learning (Inter & Intra)

To reduce prediction variance, NCL++ employs two levels of collaborative learning:

  • Inter-expert (InterCL): Multiple models, or "experts," are trained concurrently. They function as both teachers and students, utilizing Online Distillation to encourage mutual consistency in their predictions.
  • Intra-expert (IntraCL): A single model evaluates multiple augmented versions (e.g., different crops or flips) of the same image. The framework enforces consistency across these views, ensuring the model learns robust, transformation-invariant features.

2. Balanced Online Distillation

Standard knowledge distillation can inadvertently reinforce head-class bias if the teacher's predictions are skewed. NCL++ introduces a balanced probability calculation that adjusts for class frequency. This ensures that the knowledge transferred between experts does not ignore rare tail classes, facilitating a more equitable learning process across the entire distribution.

3. Hard Category Mining (HCM) and Nested Feature Learning (NFL)

The "Nested" component of the framework refers to a hierarchical approach to feature learning. The model first evaluates a Global View of all classes. It then identifies "Hard Categories"—negative classes that are most likely to be confused with the target sample. Through Nested Feature Learning (NFL), the model performs a secondary "Partial View" learning step focused specifically on distinguishing the correct class from these specific rivals. This granular focus helps the model develop a more meticulous ability to differentiate between visually similar categories.

Experimental Results

The authors evaluated NCL++ on five standard benchmarks: CIFAR-10-LT, CIFAR-100-LT, ImageNet-LT, Places-LT, and iNaturalist 2018. The latter is a naturally imbalanced dataset containing over 8,000 species and is considered the gold standard for long-tail recognition.

Performance Highlights

  • iNaturalist 2018: NCL++ achieved 74.0% accuracy with a single expert and 75.2% with a two-expert ensemble, representing a significant improvement over previous state-of-the-art (SOTA) methods.
  • ImageNet-LT: Using a standard ResNet-50 backbone, the framework reached 58.0% accuracy.
  • Efficiency: NCL++ is more efficient than its predecessor; it achieves superior performance with only two experts compared to the three experts required by the original NCL.
  • Tail Accuracy: The most notable gains were observed in the "Few" (tail) splits, where NCL++ outperformed baseline methods like Balanced Softmax Cross-Entropy (BSCE) by several percentage points.

Discussion and Conclusion

NCL++ demonstrates that treating model disagreement as a signal rather than noise can lead to more robust representations. By minimizing KL divergence between experts (InterCL) and between views (IntraCL), the framework smooths the optimization landscape for long-tailed data.

While NCL++ involves a higher training cost due to the requirement of multiple experts and augmentations, this overhead is restricted to the training phase. At inference time, a single expert can be deployed, allowing users to benefit from the collaborative training process without additional computational latency. Future research may focus on reducing the training-time overhead while maintaining the stability and accuracy gains provided by the collaborative approach.

Summary updated

Abstract

The networks trained on the long-tailed dataset vary remarkably, despite the same training settings, which shows the great uncertainty in long-tailed learning. To alleviate the uncertainty, we propose a Nested Collaborative Learning (NCL), which tackles the problem by collaboratively learning multiple experts together. NCL consists of two core components, namely Nested Individual Learning (NIL) and Nested Balanced Online Distillation (NBOD), which focus on the individual supervised learning for each single expert and the knowledge transferring among multiple experts, respectively. To learn representations more thoroughly, both NIL and NBOD are formulated in a nested way, in which the learning is conducted on not just all categories from a full perspective but some hard categories from a partial perspective. Regarding the learning in the partial perspective, we specifically select the negative categories with high predicted scores as the hard categories by using a proposed Hard Category Mining (HCM). In the NCL, the learning from two perspectives is nested, highly related and complementary, and helps the network to capture not only global and robust features but also meticulous distinguishing ability. Moreover, self-supervision is further utilized for feature enhancement. Extensive experiments manifest the superiority of our method with outperforming the state-of-the-art whether by using a single model or an ensemble. Code is available at https://github.com/Bazinga699/NCL
BibTeX
@inproceedings{li2022nested,
  author = {Jun Li and Zichang Tan and Jun Wan and Zhen Lei and Guodong Guo},
  title = {{Nested Collaborative Learning for Long-Tailed Visual Recognition}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2022},
  doi = {10.1109/cvpr52688.2022.00682},
}
Figure