Introduction
Adapting large-scale pre-trained Vision-Language Models (VLMs) like CLIP to specific downstream tasks presents a significant challenge known as the discrimination-generalization dilemma. To perform well on a new task, a model must learn task-specific features (discrimination). However, it must also retain the broad, foundational knowledge acquired during pre-training to handle unseen classes or domain shifts (generalization).
Existing adaptation strategies, such as prompt learning (e.g., CoOp) or standard adapters (e.g., CLIP-Adapter), often apply modifications uniformly across all layers or treat the model as a black box. These approaches frequently ignore the hierarchical nature of feature representation in deep networks, leading to sub-optimal alignment and a tendency to overfit on small datasets.
Motivation: Analyzing Layer-wise Discriminability
To address this dilemma, the authors conducted a "Dataset-level recognition" analysis to identify where task-specific information resides within the model. By measuring how well features from different layers could identify a specific dataset, they discovered that higher layers are significantly more discriminable than lower layers.
As shown in Figure 1, recognition accuracy for both vision and text branches increases as data progresses through the encoder. This suggests that the lower layers preserve generalizable "foundation" knowledge, while the upper layers are more susceptible to dataset-specific biases. This insight forms the empirical basis for MMA's "surgical" tuning approach.
Methodology: The Multi-Modal Adapter (MMA)
The proposed Multi-Modal Adapter (MMA) introduces a lightweight tuning strategy characterized by two main components: selective insertion and a shared multi-modal architecture.
Selective Insertion
Rather than adapting every layer, MMA selectively adds adapters only to the higher layers of both the vision and text encoders (e.g., from layer 9 through 12). By keeping the lower layers frozen, MMA protects the model's general-purpose representations while allowing the upper layers to specialize for the target task.
The Multi-Modal Unit
Unlike standard unimodal adapters, MMA employs a unique "Down-Shared-Up" architecture. For each adapted layer:
- Down: Independent layers reduce the dimensionality of the vision and text features.
- Shared: A central projection layer is shared by both modalities. This acts as a gradient bridge, forcing the vision and text branches to align their task-specific representations during training.
- Up: Features are projected back to their original dimensions and added to the transformer output via a scaled residual connection ($\alpha$).
Experimental Results
MMA was evaluated across a comprehensive suite of benchmarks, including 11 diverse datasets such as ImageNet, StanfordCars, and EuroSAT. It consistently outperformed state-of-the-art methods like MaPLe and CoOp across three key evaluation metrics:
- Base-to-Novel Generalization: MMA achieved a leading average Harmonic Mean (HM) of 79.87% across 11 datasets, surpassing the previous best (MaPLe) by approximately 1.3 percentage points. This confirms that MMA effectively balances task learning with the ability to generalize to unseen classes.
- Cross-Dataset Transfer: When trained on ImageNet and tested on 10 other datasets, MMA reached an average accuracy of 66.61%, demonstrating superior zero-shot transferability compared to prompt-based methods.
- Domain Generalization: On out-of-distribution variants of ImageNet (V2, Sketch, A, and R), MMA showed the best robustness in 3 out of 4 scenarios, proving that selective adaptation does not compromise the model's inherent robustness.
Conclusion
MMA demonstrates that a "less is more" approach to fine-tuning can yield superior results for Vision-Language Models. By targeting only the most discriminable layers and utilizing a shared multi-modal bottleneck, MMA successfully navigates the trade-off between task-specific performance and broad generalization. Future research may explore automated methods for layer selection and extending this multi-modal unit to incorporate additional data types like audio or depth information.
Summary updated