Abstract
Real-world data is predominantly unbalanced and long-tailed, but deep models struggle to recognize rare classes in the presence of frequent classes. Often, classes can be accompanied by side information like textual descriptions, but it is not fully clear how to use them for learning with unbalanced long-tail data. Such descriptions have been mostly used in (Generalized) Zero-shot learning (ZSL), suggesting that ZSL with class descriptions may also be useful for long-tail distributions. We describe DRAGON, a late-fusion architecture for long-tail learning with class descriptors. It learns to (1) correct the bias towards head classes on a sample-by-sample basis; and (2) fuse information from class-descriptions to improve the tail-class accuracy. We also introduce new benchmarks CUB-LT, SUN-LT, AWA-LT for long-tail learning with class-descriptions, building on existing learning-with-attributes datasets and a version of Imagenet-LT with class descriptors. DRAGON outperforms state-of-the-art models on the new benchmark. It is also a new SoTA on existing benchmarks for GFSL with class descriptors (GFSL-d) and standard (vision-only) long-tailed learning ImageNet-LT, CIFAR-10, 100, and Places365.
Introduction
In the real world, data is rarely distributed evenly. Most datasets follow a "long-tail" distribution: a few "head" classes are seen constantly, while a vast "tail" of rare classes is seldom encountered. Deep learning models struggle in these environments due to two primary issues: data paucity in the tail and a pronounced "familiarity bias." This bias causes models to become overconfident in frequent classes, showing high (and often unearned) confidence in "head" predictions while neglecting the "tail."
While side information like textual descriptions or attributes is commonly used in Zero-Shot Learning (ZSL), this paper explores how to use that same information to bridge the gap in long-tail scenarios where some training data exists, but it is extremely scarce. The authors introduce DRAGON (moDulaR Approach for lonG-tail classificatiON), a novel late-fusion architecture that tackles this problem by combining visual data with textual class descriptors.
Methods: The DRAGON Architecture
DRAGON is a modular framework consisting of three main parts designed to balance visual recognition with semantic knowledge:
- The Visual Expert: A standard CNN (like ResNet) trained on images. It is effective at identifying "head" classes but struggles with the "tail."
- The Semantic Expert: A module that uses class descriptors (attributes or word embeddings). Because semantic descriptions are shared across classes, this expert is much better at generalizing to rare classes.
- The Fusion Module: Instead of simple averaging, DRAGON uses a sophisticated late-fusion approach. It reorders class predictions based on their scores to utilize "order statistics"—the gap between the 1st and 2nd most likely prediction—as a signal for true confidence. Crucially, it uses a small convolutional network and polynomial regression to learn a specific weight for each prediction on a per-sample basis, rather than a per-class basis.
The authors also present smDRAGON, a "vision-only" version that applies this debiasing logic to a single visual model, proving the rebalancing technique works even without text descriptors.
Datasets and Key Results
The researchers evaluated DRAGON across several challenging scenarios, including new "Smooth-Tail" benchmarks (CUB-LT, SUN-LT, and AWA-LT) and standard vision-only benchmarks like ImageNet-LT, CIFAR-10/100, and iNaturalist 2018.
DRAGON consistently outperformed existing state-of-the-art (SoTA) methods:
- On CUB-LT: DRAGON achieved 60.1% per-class accuracy, significantly beating the previous best multi-modal approach (CADA-VAE at 48.3%).
- On ImageNet-LT: The vision-only variant (smDRAGON) achieved 42.0% accuracy (ResNet-10), outperforming standard long-tail methods like OLTR (35.6%) and Classifier Balancing (41.4%).
Discussion and Insights
A key insight of this work is that "Familiarity Bias" is measurable: models are not just more accurate on head classes; they are significantly more overconfident. Traditional methods often penalize head classes across the board, but DRAGON realizes that "easy" prototypical samples of a head class deserve high confidence. By sorting predictions before processing them, the model becomes "rank-aware," learning that a 0.6 confidence score means something very different if the second-best guess is 0.59 versus 0.1.
While DRAGON is powerful, it currently relies on high-quality class descriptors. Future work could explore automatically generating these descriptors from raw web text to make the multi-modal approach more scalable across diverse domains.
Summary updated