TL;DR
This paper introduces HomoGraphAdapter, a novel method for adapting pre-trained Vision-Language Models (VLMs) like CLIP to new tasks. It leverages a homogeneous graph neural network to learn from both textual and visual information, achieving better accuracy and efficiency in few-shot learning scenarios compared to existing graph-based methods.
What Problem Does This Paper Address?
Adapting powerful pre-trained Vision-Language Models (VLMs) such as CLIP to downstream tasks, especially with limited labeled data (few-shot learning), remains a significant challenge. Existing methods often either ignore the structural relationships between text and image data or create overly complex graph structures that lead to suboptimal performance and increased computational costs. Specifically, previous graph-based approaches have either treated text and image modalities separately or used heterogeneous graphs that might introduce unnecessary complexity for the inherently dual-modality nature of CLIP. This paper aims to bridge this gap by proposing a more efficient and effective graph-based adaptation strategy.
Methods
The core innovation is HomoGraphAdapter, which constructs a homogeneous graph to represent textual and visual embeddings. Here's how it works:
- Unified Node Representation: Diverse textual descriptions (general, detailed, negative) and visual features from few-shot examples are treated as nodes of the same type within a single graph.
- Homogeneous Graph Learning: A single-layer homogeneous Graph Convolutional Network (GCN) is employed to process this graph. This GCN learns to refine node features by aggregating information from their neighbors, effectively capturing semantic alignments.
- Edge Connections: Edges are established between nodes that exhibit semantic alignment, focusing on intra-class and cross-modal relationships. Crucially, HomoGraphAdapter only uses edge connections (index pairs) and eliminates the need for pre-computed edge features, simplifying the process and reducing redundancy.
- Classifier Adaptation: The graph learning process adapts both positive and negative classifiers for text and image modalities, leading to improved few-shot classification and out-of-distribution generalization.
- Logit Combination: Finally, logits from multiple sources (positive text, negative text, positive visual, negative visual) are combined to produce the final prediction.
Datasets
The paper evaluates HomoGraphAdapter on 11 benchmark image classification datasets:
- ImageNet
- StanfordCars
- UCF101
- Caltech101
- Flowers102
- SUN397
- DTD
- EuroSAT
- FGVCAircraft
- OxfordPets
- Food101
These datasets cover a range of tasks including remote sensing, action recognition, texture classification, and fine-grained classification.
Key Insights and Ideas
- Simplicity is Key: By using a homogeneous graph and only edge connections, HomoGraphAdapter simplifies the graph construction process compared to previous heterogeneous graph approaches, leading to better efficiency without sacrificing performance.
- Unified Modality Learning: Treating text and image embeddings as nodes of the same type within a single graph allows for more direct and effective cross-modal learning.
- Leveraging Negative Information: The explicit incorporation of negative textual and visual classifiers, inspired by recent work, significantly boosts performance, especially in challenging scenarios.
- Efficiency Gains: The method achieves competitive or superior accuracy while requiring less pre-computation and training time, making it more practical for real-world applications.
Key Results
- Accuracy Improvement: HomoGraphAdapter outperforms state-of-the-art graph-based adapter methods (GraphAdapter and HeGraphAdapter) by an average of 1.51% in 1-shot settings and 0.74% in 16-shot settings across the 11 datasets.
- Efficiency: It achieves the shortest pre-computation time among graph-based methods and requires only 12.6 seconds per epoch for training, with a total of 2.07 million trainable parameters on ImageNet.
- Robustness: The model demonstrates strong performance across various CLIP backbones (ResNet-50, ResNet-101, ViT-B/32, ViT-B/16) and on Out-Of-Distribution (OOD) datasets like ImageNet-V2 and ImageNet-Sketch.
- Ablation Studies: The paper shows that each component (positive/negative text classifiers, positive/negative visual caches) contributes positively to the overall performance.
Limitations and Future Work
- CLIP's Intrinsic Limitations: The paper acknowledges that the performance enhancement from negative classifiers is inherently limited by CLIP's original training, which did not include negative descriptions. This might impede the model's ability to effectively differentiate between very similar or ambiguous inputs.
- Task-Specific Adaptation: Like most adapter-tuning methods, HomoGraphAdapter is fine-tuned for a specific downstream task and cannot be directly applied to another task without re-adaptation.
- Future Directions: The authors suggest exploring the integration of negative learning more deeply with positive learning to achieve more nuanced representations. They also aim to improve the robustness of adapter-tuning approaches for better generalization across diverse conditions and to extend graph learning to other pre-trained multi-modality models.
Summary updated