From generalized zero-shot learning to long-tail with class descriptors

Dvir Samuel, Yuval Atzmon, Gal Chechik

Introduction: The Long-Tail Problem

Real-world datasets often exhibit a "long-tail" distribution, where a small number of "head" classes possess the majority of training samples, while a vast number of "tail" classes have very few. Standard deep learning models struggle in this regime due to two primary factors: data paucity (insufficient visual examples for tail classes) and familiarity bias (a tendency to misclassify rare objects as common ones due to overexposure during training).

The long-tail distribution (a) leads to a significant drop in average confidence for tail classes (b) and overall model miscalibration (c).

While class descriptors (textual attributes) have traditionally been used in Zero-Shot Learning (ZSL) to recognize unseen categories, this paper proposes leveraging such side-information to mitigate the imbalances inherent in long-tail recognition.

The DRAGON Architecture

The authors introduce DRAGON, a modular late-fusion architecture designed to bridge the gap between generalized zero-shot learning and long-tail recognition. The system is composed of three primary components:

  1. Visual Expert: A standard convolutional neural network (e.g., ResNet) trained on visual data. While proficient at identifying head classes, its performance degrades significantly on the tail.
  2. Semantic Expert: A classifier that utilizes class descriptors (e.g., "has a red beak"). By relying on semantic definitions rather than sample frequency, it maintains better discriminative power for rare classes.
  3. Fusion & Debiasing Module: This module integrates the experts' outputs through a novel per-sample debiasing technique. It utilizes order statistics (sorting predictions by confidence) to gauge model certainty and applies learned weights based on class frequency to correct familiarity bias.
The DRAGON architecture fuses a Visual Expert and a Semantic Expert through a dedicated Fusion Module that incorporates per-sample debiasing.

Experimental Evaluation

DRAGON was evaluated on several benchmarks, including newly created "smooth-tail" versions of attribute-rich datasets: CUB-LT, SUN-LT, and AWA-LT. It was also tested on ImageNet-LT-d, which augments the standard ImageNet-LT with word2vec descriptors.

Key Results

  • State-of-the-Art Performance: On ImageNet-LT-d, DRAGON achieved 51.2% per-class accuracy, outperforming multi-modal baselines like CADA-VAE.
  • Rare Class Improvement: In the CUB-LT dataset, DRAGON reached 60.1% accuracy, a significant jump from the 53.0% achieved by standard cross-entropy models.
  • Universal Debiasing: Even without textual descriptors, the debiasing module alone improved performance on vision-only datasets like Places365-LT, demonstrating its effectiveness in correcting familiarity bias.

Conclusion and Future Work

DRAGON demonstrates that the challenges of long-tail recognition can be effectively addressed by treating tail classes as a few-shot or zero-shot problem. By fusing visual and semantic experts with a sophisticated debiasing mechanism, the model achieves superior performance across diverse imbalanced datasets. Future research may explore "mid-fusion" strategies to allow for shared feature learning between modalities and methods to reduce the computational overhead of running dual expert streams.

Summary updated

Abstract

Real-world data is predominantly unbalanced and long-tailed, but deep models struggle to recognize rare classes in the presence of frequent classes. Often, classes can be accompanied by side information like textual descriptions, but it is not fully clear how to use them for learning with unbalanced long-tail data. Such descriptions have been mostly used in (Generalized) Zero-shot learning (ZSL), suggesting that ZSL with class descriptions may also be useful for long- tail distributions.We describe Dragon, a late-fusion architecture for long-tail learning with class descriptors. It learns to (1) correct the bias towards head classes on a sample- by-sample basis; and (2) fuse information from class- descriptions to improve the tail-class accuracy. We also introduce new benchmarks CUB-LT, SUN-LT, AWA-LT for long-tail learning with class-descriptions, building on existing learning-with-attributes datasets and a version of Imagenet-LT with class descriptors. Dragon outperforms state-of-the-art models on the new benchmark. It is also a new SoTA on existing benchmarks for GFSL with class descriptors (GFSL-d) and standard (vision-only) long-tailed learning ImageNet-LT, CIFAR-10, 100, and Places365-LT.
BibTeX
@inproceedings{samuel2021from,
  author = {Dvir Samuel and Yuval Atzmon and Gal Chechik},
  title = {{From generalized zero-shot learning to long-tail with class descriptors}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2021},
  doi = {10.1109/wacv48630.2021.00033},
}
Figure