Long-Tailed Recognition via Information-Preservable Two-Stage Learning

Fudong Lin, Xu Yuan

Introduction

In real-world applications, datasets rarely follow a uniform distribution. Instead, they typically exhibit a "long-tailed" distribution where a small number of "head" classes contain the majority of samples, while many "tail" classes are represented by only a few examples. Standard deep neural networks (DNNs) trained on such data tend to develop decision boundaries heavily biased toward head classes, leading to poor performance on rare but often critical categories—such as rare diseases in medical imaging or infrequent obstacles in autonomous driving.

To address this, researchers frequently employ two-stage learning: first training a feature extractor to learn robust representations, and then fine-tuning the classifier on a balanced subset of data. However, existing methods often fall short. In the first stage, they struggle to create well-separated feature spaces for tail classes. In the second stage, they rely on flawed sampling strategies: oversampling can lead to model overfitting (mode collapse), while random undersampling results in significant information loss. This paper introduces a novel framework designed to rectify these bottlenecks through information-theoretic representation learning and diversity-aware sampling.

Methodology

The proposed framework optimizes the two-stage pipeline using two core components: Balanced Negative Sampling (BNS) and Information-Preservable Determinantal Point Process (IP-DPP).

Stage 1: Balanced Negative Sampling (BNS)

Rather than relying on standard contrastive learning, BNS approaches representation learning from an information theory perspective. For any given anchor image, the model samples additional images from the same class. The objective function is designed to maximize the mutual information between these augmented views.

The authors provide a formal mathematical proof demonstrating that maximizing this mutual information is equivalent to minimizing intra-class distance. This forcing function encourages the model to learn dual-level semantics:

  1. Instance-level semantics: What makes a specific image unique.
  2. Class-level semantics: What features are shared across the entire category.

This results in a feature space where clusters are tighter and more distinct, even for classes with limited representation.

Stage 2: Information-Preservable Determinantal Point Process (IP-DPP)

To fine-tune the classifier, the model requires a balanced dataset. To avoid the information loss associated with random undersampling, the authors utilize a Determinantal Point Process (DPP). A DPP is a stochastic process that captures global negative correlations, naturally preferring the selection of diverse, non-redundant items.

The authors refined this into IP-DPP, which prioritizes "informative" instances—specifically those the model currently finds difficult to classify—while enforcing a fixed sample size to ensure perfect class balance. This approach demonstrates that the quality and diversity of the retained data are more critical for performance than the sheer volume of data.

Experimental Results

The framework was evaluated against several competitive benchmarks, including CIFAR-10-LT, CIFAR-100-LT, ImageNet-LT, and the naturally long-tailed iNaturalist 2018 dataset.

  • CIFAR-10-LT: The method achieved 76.4% overall accuracy, outperforming the previous state-of-the-art (OTmix) by 2.6%. Notably, it improved few-shot (tail class) accuracy by nearly 20%.
  • CIFAR-100-LT: The model reached 52.4% accuracy, a 3.2% improvement over the prior SOTA (DisA).
  • Large-Scale Performance: On ImageNet-LT and iNaturalist 2018, the method achieved 51.7% and 74.0% accuracy, respectively, proving its scalability to massive, real-world datasets.
  • Representation Quality: In linear probing tests (where the feature extractor is frozen), BNS achieved 68.2% on CIFAR-10-LT, significantly higher than other contrastive methods like KCL (61.0%) and SBCL (64.7%).

Efficiency and Modularity

Despite the mathematical complexity of the Determinantal Point Process, the authors implemented an efficient sampling algorithm. On CIFAR-10-LT, the overhead was minimal—only 0.8 seconds slower than random sampling—and the overall training time was actually reduced because the model trains on a smaller, more intelligent subset of the data.

Conclusion

This research demonstrates that long-tailed recognition can be significantly improved by focusing on information preservation. By combining BNS for tighter feature clustering and IP-DPP for diverse, informative sampling, the framework achieves state-of-the-art results without the typical pitfalls of data loss or overfitting. Furthermore, the modular nature of these components allows them to be integrated into existing machine learning pipelines to enhance performance across various imbalanced data domains.

Summary updated

Abstract

The imbalance (or long-tail) is the nature of many real-world data distributions, which often induces the undesirable bias of deep classification models toward frequent classes, resulting in poor performance for tail classes. In this paper, we propose a novel two-stage learning approach to mitigate such a majority-biased tendency while preserving valuable information within datasets. Specifically, the first stage proposes a new representation learning technique from the information theory perspective. This approach is theoretically equivalent to minimizing intra-class distance, yielding an effective and well-separated feature space. The second stage develops a novel sampling strategy that selects mathematically informative instances, able to rectify majority-biased decision boundaries without compromising a model’s overall performance. As a result, our approach achieves the state-of-the-art performance across various long-tailed benchmark datasets, validated via extensive experiments. Our code is available at https://github.com/fudong03/BNS_IPDPP.
BibTeX
@inproceedings{lin2025long,
  author = {Fudong Lin and Xu Yuan},
  title = {{Long-Tailed Recognition via Information-Preservable Two-Stage Learning}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2025},
  url = {https://openreview.net/forum?id=UBsYf2lyNE},
}
Figure