Abstract
Real-world data is predominantly unbalanced and long-tailed, but deep models struggle to recognize rare classes in the presence of frequent classes. Often, classes can be accompanied by side information like textual descriptions, but it is not fully clear how to use them for learning with unbalanced long-tail data. Such descriptions have been mostly used in (Generalized) Zero-shot learning (ZSL), suggesting that ZSL with class descriptions may also be useful for long-tail distributions. We describe DRAGON, a late-fusion architecture for long-tail learning with class descriptors. It learns to (1) correct the bias towards head classes on a sample-by-sample basis; and (2) fuse information from class-descriptions to improve the tail-class accuracy. We also introduce new benchmarks CUB-LT, SUN-LT, AWA-LT for long-tail learning with class-descriptions, building on existing learning-with-attributes datasets and a version of Imagenet-LT with class descriptors. DRAGON outperforms state-of-the-art models on the new benchmark. It is also a new SoTA on existing benchmarks for GFSL with class descriptors (GFSL-d) and standard (vision-only) long-tailed learning ImageNet-LT, CIFAR-10, 100, and Places365.
Introduction: The Challenge of Familiarity Bias
In real-world scenarios, data distribution is rarely uniform. Most datasets exhibit a "long-tail" distribution where a few "head" classes have abundant samples, while the vast majority of "tail" classes are represented by only a few examples. Standard deep learning models trained on such data suffer from a significant performance drop on tail classes due to two primary factors: data paucity and familiarity bias.
Familiarity bias occurs when models become overconfident in frequent head classes. They don't just predict these classes more often; they do so with disproportionately high confidence, effectively "crowding out" correct predictions for rare classes during the final classification stage. While side information like textual descriptions or attributes is often available, effectively integrating this semantic knowledge to mitigate long-tail imbalance has remained a challenge.
The DRAGON Architecture
To address these issues, we introduce DRAGON (a moDulaR Approach for lonG-tail classificatiON). DRAGON utilizes a late-fusion strategy to combine visual features with semantic class descriptors.
The architecture consists of three core components:
- Visual Expert: A standard CNN (e.g., ResNet) that extracts visual features. While proficient at head classes, its performance degrades on the tail.
- Semantic Expert: This module predicts classes based on semantic descriptors. Because attributes (like "has a red beak") are often shared across both head and tail classes, this expert provides a more robust signal for rare categories.
- Fusion Module: This component dynamically re-balances predictions. It sorts class predictions by confidence to identify patterns in the "confidence gaps" and uses a polynomial regression based on training sample counts ($n_y$) to scale scores. This process effectively "cools down" overconfident head-class predictions on a sample-by-sample basis.
For scenarios where class descriptors are unavailable, a simplified variant called smDRAGON applies the same debiasing logic to a single vision-only model.
Experimental Evaluation
We evaluated DRAGON across several new and existing benchmarks. We introduced CUB-LT, SUN-LT, and AWA-LT—long-tailed versions of popular attribute-rich datasets. Additionally, we tested on ImageNet-LT (augmented with descriptors) and standard vision-only benchmarks like CIFAR-100-LT and Places365-LT.
DRAGON consistently achieved state-of-the-art (SoTA) results:
- ImageNet-LT: Achieved 53.5% accuracy, surpassing the previous best of 49.9%.
- Attribute Datasets: On CUB-LT, DRAGON reached 60.1% accuracy, a significant improvement over the 53.0% vision-only baseline.
- Vision-Only: Even without attributes,
smDRAGONset new SoTA records on CIFAR-100-LT and Places365-LT, demonstrating the intrinsic value of the debiasing module.
Key Insights and Conclusion
A central finding of this work is that the profile of the prediction vector—specifically the gap between top scores—is a critical signal for calibration. Unlike traditional methods that penalize head classes uniformly, DRAGON learns to penalize them only when the model appears suspiciously overconfident for a specific image. By treating the tail classes as a few-shot learning problem and leveraging semantic "side-loading," DRAGON provides a principled framework for robust long-tail recognition. Future work will focus on streamlining the multi-stage training process into an end-to-end pipeline and optimizing the fusion module for low-latency applications.
Summary updated