TL;DR
This paper introduces TASA, a novel framework for long-tailed image classification that tackles the challenge of underrepresented "tail" classes. TASA uses LLM-generated descriptions to create stable text prototypes for each class and employs a new cross-modal fusion module (DCF) to better align image and text features, leading to significant improvements, especially for rare categories.
What Problem Does This Paper Address?
Real-world image datasets often exhibit a "long-tailed" distribution, meaning a few classes (head classes) have abundant data, while many others (tail classes) have very few samples. This imbalance makes it difficult for models to learn effectively, especially for the tail classes, leading to poor generalization. While vision-language models like CLIP have shown promise by leveraging text, simple text prompts often provide weak supervision, exacerbating the bias towards head classes. Existing methods struggle to provide sufficient guidance for tail classes when visual data is scarce, and prompt-tuning approaches can be data-hungry and prone to overfitting.
Methods
TASA is an end-to-end framework designed to stabilize textual supervision and enhance cross-modal fusion. It comprises three main components:
- Semantic Distribution Modulation (SDM): This module addresses the sparsity and noise in tail classes by generating class-specific text prototypes. It leverages Large Language Models (LLMs) to create diverse descriptions for each class, which are then fused with a canonical template (e.g., "a photo of a [CLASS]") using cosine similarity weighting. This creates stable, semantically rich anchors without requiring text parameter training.
- Dual-Space Cross-Modal Fusion (DCF): This novel module enables bidirectional interaction between image and text features. It uses selective-scan state-space blocks in both image and text branches, allowing them to condition each other. This mechanism facilitates efficient feature fusion and improves the separability of tail categories through dynamic importance reweighting.
- Margin-Aware Alignment Loss (MMS): This loss function, based on Margin Metric Softmax, aligns image features with their corresponding text prototypes. It augments standard softmax by incorporating prototype distances, effectively enlarging margins against semantically distant negative classes and improving tail discrimination.
Datasets
The proposed TASA framework was evaluated on several standard long-tailed image classification benchmarks:
- CIFAR-10-LT and CIFAR-100-LT: Variants of the CIFAR datasets with imbalanced class distributions.
- ImageNet-LT: A long-tailed version of the ImageNet dataset.
- Places-LT: A long-tailed dataset for scene classification.
The paper also mentions using Flickr30k for image-text retrieval experiments.
Key Insights and Ideas
- LLMs as a source of rich, diverse textual supervision: Instead of relying on simple, potentially biased prompts, TASA effectively harnesses LLMs to generate a richer semantic understanding of classes, which is crucial for tail classes.
- Text prototypes as stable anchors: The SDM module's approach of creating class-specific text prototypes from LLM descriptions provides robust semantic anchors that guide the model, especially when visual evidence is limited.
- State-space models for efficient cross-modal interaction: The DCF module's use of selective-scan state-space models allows for efficient, bidirectional conditioning between image and text features, leading to improved fusion and better handling of imbalanced data.
- Complementarity of SDM and DCF: Ablation studies reveal that while DCF offers the largest single-module gain, the combination of SDM and DCF provides the most robust and balanced performance, highlighting their synergistic effect.
Key Results
TASA consistently outperformed strong baselines, including CLIP-based methods, across various long-tailed datasets and imbalance ratios.
- CIFAR-100-LT: TASA achieved top-1 accuracies of 79.4%, 80.6%, and 84.4% for imbalance ratios of 100, 50, and 10, respectively, outperforming the MMS baseline by significant margins.
- CIFAR-10-LT: TASA achieved top-1 accuracies of 94.1%, 94.9%, and 94.6% for imbalance ratios of 100, 50, and 10.
- ImageNet-LT and Places-LT: TASA also demonstrated superior performance on these more complex datasets, showing the generalizability of the approach across different visual domains.
- Grouped Results (CIFAR-100-LT): TASA showed notable improvements in few-shot and medium-shot groups, indicating its effectiveness in addressing the challenges posed by rare classes.
- Component Analysis: DCF provided an average improvement of 2.5 points, SDM 1.5 points, and their combination 3.2 points, with the largest gains observed under severe imbalance (IR=100).
- Image-Text Retrieval (Flickr30k): Even as a training-free reranker, DCF showed consistent improvements in Recall@1 for both text-to-image and image-to-text retrieval.
Limitations and Future Work
- CLIP Bias: The authors acknowledge that using CLIP text anchors for prototype construction might introduce a mild bias towards CLIP's embedding space. However, they note that this influence is limited due to the adaptive weighting of the frozen prior.
- Computational Overhead: The DCF module introduces additional computation and latency compared to simpler contrastive baselines.
- Future Directions: The paper suggests several avenues for future research:
- Extending TASA to other vision tasks like detection, segmentation, and open-vocabulary retrieval.
- Developing adaptive description generation strategies.
- Exploring lightweight fusion designs, such as attention-state-space hybrids.
- Conducting theoretical analysis of prototype variance and alignment margins to enhance robustness in extreme long-tailed settings.
Summary updated