Long-tail learning with class descriptors

Dvir Samuel, Yuval Atzmon, Gal Chechik

Abstract

Real-world data is predominantly unbalanced and long-tailed, but deep models struggle to recognize rare classes in the presence of frequent classes. Often, classes can be accompanied by side information like textual descriptions, but it is not fully clear how to use them for learning with unbalanced long-tail data. Such descriptions have been mostly used in (Generalized) Zero-shot learning (ZSL), suggesting that ZSL with class descriptions may also be useful for long-tail distributions. We describe DRAGON, a late-fusion architecture for long-tail learning with class descriptors. It learns to (1) correct the bias towards head classes on a sample-by-sample basis; and (2) fuse information from class-descriptions to improve the tail-class accuracy. We also introduce new benchmarks CUB-LT, SUN-LT, AWA-LT for long-tail learning with class-descriptions, building on existing learning-with-attributes datasets and a version of Imagenet-LT with class descriptors. DRAGON outperforms state-of-the-art models on the new benchmark. It is also a new SoTA on existing benchmarks for GFSL with class descriptors (GFSL-d) and standard (vision-only) long-tailed learning ImageNet-LT, CIFAR-10, 100, and Places365.

Introduction: The Long-Tail Challenge

In the real world, data distributions are rarely balanced. A few categories appear frequently (the "head"), while the vast majority are rare (the "tail"). Standard deep learning models tend to over-optimize for head classes, often at the expense of rare ones. This "long-tail" problem is a significant hurdle for deploying AI in wild, uncurated environments.

The paper introduces DRAGON (moDulaR Approach for lonG-tail classificatiON), a late-fusion architecture that leverages side information—such as textual descriptions or attributes—to improve recognition of rare classes.

The Problem: Familiarity Bias and Overconfidence

Deep learning models struggle with long-tailed data for two primary reasons. First, there is a simple lack of visual examples for rare classes. Second, models develop a "familiarity bias." Because the model encounters head classes far more often, it becomes biased toward them.

Crucially, the authors demonstrate that models are not just more accurate on head classes; they are significantly overconfident. This overconfidence creates a "shouting" effect where the vision model's biased predictions drown out potentially more accurate "whispers" from other modules or experts.

Familiarity bias in long-tail learning: (a) class distribution, (b) average confidence per class, and (c) reliability diagram showing the gap between confidence and accuracy.

The DRAGON Architecture

DRAGON is designed as a modular late-fusion system built on three key pillars:

  1. Dual Experts: The system employs a Visual Expert (typically a standard CNN) and a Semantic Expert. The Semantic Expert utilizes class descriptors (attributes or word embeddings) to capture relationships between classes, providing a prior that is less sensitive to visual frequency.
  2. Fusion Module: Instead of simple averaging, DRAGON uses a dedicated fusion module. It stacks predictions from both experts, sorts them by confidence to focus on relative differences, and processes them through a small convolutional network.
  3. Dynamic Debiasing: To combat familiarity bias, the authors include a polynomial regression component. This module takes the number of training samples per class ($n_y$) as input and learns to rescale predictions dynamically. This effectively "muffles" overconfident head classes and "amplifies" tail classes on a per-sample basis.
The DRAGON architecture: A late-fusion approach combining visual and semantic experts with a dynamic debiasing module.

For cases where class descriptors are unavailable, a single-modality variant called smDRAGON applies the same debiasing logic to standard vision-only tasks.

New Benchmarks and Evaluation

To rigorously test long-tail learning with side information, the authors introduced three new benchmarks: CUB-LT (birds), SUN-LT (scenes), and AWA-LT (animals), all derived from existing attribute-rich datasets but modified to follow long-tail distributions. They also evaluated on standard benchmarks like ImageNet-LT, CIFAR-10/100, and Places365-LT.

Key Results and Insights

DRAGON achieves state-of-the-art (SoTA) performance across the board:

  • ImageNet-LT: Achieved a per-class accuracy of 53.5% (with Balanced Loss), significantly outperforming previous methods like OLTR (37.7%) and LWS (49.9%).
  • Multi-modal Gains: On CUB-LT and SUN-LT, DRAGON improved accuracy by 3-5% over existing multi-modal models.
  • Vision-Only Success: Even without descriptors, smDRAGON set new SoTA records on CIFAR and Places365, proving the debiasing module's general utility.

The research highlights that overconfidence is a signal. By treating bias as a measurable quantity related to class frequency, it can be systematically corrected. Furthermore, the success of late fusion suggests that modularity allows for better integration of diverse data types without the need for complex end-to-end retraining.

Conclusion

DRAGON provides a robust framework for long-tail learning by addressing the root cause of model bias: overconfidence in frequent classes. By combining visual data with semantic descriptors and applying dynamic debiasing, it sets a new standard for performance on unbalanced datasets. Future work may explore earlier information sharing between experts to further refine feature representations while maintaining the benefits of modularity.

Summary updated

Abstract

Real-world data is predominantly unbalanced and long-tailed, but deep models struggle to recognize rare classes in the presence of frequent classes. Often, classes can be accompanied by side information like textual descriptions, but it is not fully clear how to use them for learning with unbalanced long-tail data. Such descriptions have been mostly used in (Generalized) Zero-shot learning (ZSL), suggesting that ZSL with class descriptions may also be useful for long-tail distributions. We describe DRAGON, a late-fusion architecture for long-tail learning with class descriptors. It learns to (1) correct the bias towards head classes on a sample-by-sample basis; and (2) fuse information from class-descriptions to improve the tail-class accuracy. We also introduce new benchmarks CUB-LT, SUN-LT, AWA-LT for long-tail learning with class-descriptions, building on existing learning-with-attributes datasets and a version of Imagenet-LT with class descriptors. DRAGON outperforms state-of-the-art models on the new benchmark. It is also a new SoTA on existing benchmarks for GFSL with class descriptors (GFSL-d) and standard (vision-only) long-tailed learning ImageNet-LT, CIFAR-10, 100, and Places365.
BibTeX
@inproceedings{samuel2021long,
  author = {Dvir Samuel and Yuval Atzmon and Gal Chechik},
  title = {{Long-tail learning with class descriptors}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2021},
  doi = {10.1109/wacv48630.2021.00033},
}
Figure