MMA: Multi-Modal Adapter for Vision-Language Models

Lingxiao Yang, Ru-Yuan Zhang, Yanchen Wang, Xiaohua Xie

Introduction

Adapting large-scale pre-trained Vision-Language Models (VLMs) like CLIP to specific downstream tasks presents a significant challenge known as the discrimination-generalization dilemma. To perform well on a new task, a model must learn task-specific features (discrimination). However, it must also retain the broad, foundational knowledge acquired during pre-training to handle unseen classes or domain shifts (generalization).

Existing adaptation strategies, such as prompt learning (e.g., CoOp) or standard adapters (e.g., CLIP-Adapter), often apply modifications uniformly across all layers or treat the model as a black box. These approaches frequently ignore the hierarchical nature of feature representation in deep networks, leading to sub-optimal alignment and a tendency to overfit on small datasets.

Motivation: Analyzing Layer-wise Discriminability

To address this dilemma, the authors conducted a "Dataset-level recognition" analysis to identify where task-specific information resides within the model. By measuring how well features from different layers could identify a specific dataset, they discovered that higher layers are significantly more discriminable than lower layers.

Figure 1: Dataset-level recognition accuracy across encoder layers. The data shows that higher layers capture more dataset-specific information, while lower layers remain more general.

As shown in Figure 1, recognition accuracy for both vision and text branches increases as data progresses through the encoder. This suggests that the lower layers preserve generalizable "foundation" knowledge, while the upper layers are more susceptible to dataset-specific biases. This insight forms the empirical basis for MMA's "surgical" tuning approach.

Methodology: The Multi-Modal Adapter (MMA)

The proposed Multi-Modal Adapter (MMA) introduces a lightweight tuning strategy characterized by two main components: selective insertion and a shared multi-modal architecture.

Selective Insertion

Rather than adapting every layer, MMA selectively adds adapters only to the higher layers of both the vision and text encoders (e.g., from layer 9 through 12). By keeping the lower layers frozen, MMA protects the model's general-purpose representations while allowing the upper layers to specialize for the target task.

The Multi-Modal Unit

Unlike standard unimodal adapters, MMA employs a unique "Down-Shared-Up" architecture. For each adapted layer:

  1. Down: Independent layers reduce the dimensionality of the vision and text features.
  2. Shared: A central projection layer is shared by both modalities. This acts as a gradient bridge, forcing the vision and text branches to align their task-specific representations during training.
  3. Up: Features are projected back to their original dimensions and added to the transformer output via a scaled residual connection ($\alpha$).
Figure 2: The MMA architecture. It features selective adapter insertion in higher layers and a shared projection layer to facilitate cross-modal alignment.

Experimental Results

MMA was evaluated across a comprehensive suite of benchmarks, including 11 diverse datasets such as ImageNet, StanfordCars, and EuroSAT. It consistently outperformed state-of-the-art methods like MaPLe and CoOp across three key evaluation metrics:

  • Base-to-Novel Generalization: MMA achieved a leading average Harmonic Mean (HM) of 79.87% across 11 datasets, surpassing the previous best (MaPLe) by approximately 1.3 percentage points. This confirms that MMA effectively balances task learning with the ability to generalize to unseen classes.
  • Cross-Dataset Transfer: When trained on ImageNet and tested on 10 other datasets, MMA reached an average accuracy of 66.61%, demonstrating superior zero-shot transferability compared to prompt-based methods.
  • Domain Generalization: On out-of-distribution variants of ImageNet (V2, Sketch, A, and R), MMA showed the best robustness in 3 out of 4 scenarios, proving that selective adaptation does not compromise the model's inherent robustness.

Conclusion

MMA demonstrates that a "less is more" approach to fine-tuning can yield superior results for Vision-Language Models. By targeting only the most discriminable layers and utilizing a shared multi-modal bottleneck, MMA successfully navigates the trade-off between task-specific performance and broad generalization. Future research may explore automated methods for layer selection and extending this multi-modal unit to incorporate additional data types like audio or depth information.

Summary updated

Abstract

Pretrained Vision-Language Models (VLMs) have served as excellent foundation models for transfer learning in diverse downstream tasks. However, tuning VLMs for few-shot generalization tasks faces a discrimination - generalization dilemma, i.e., general knowledge should be preserved and task-specific knowledge should be fine-tuned. How to precisely identify these two types of representations remains a challenge. In this paper, we propose a Multi-Modal Adapter (MMA) for VLMs to improve the alignment between representations from text and vision branches. MMA aggregates features from different branches into a shared feature space so that gradients can be communicated across branches. To determine how to incorporate MMA, we systematically analyze the discriminability and generalizability of features across diverse datasets in both the vision and language branches, and find that (1) higher lay-ers contain discriminable dataset-specific knowledge, while lower layers contain more generalizable knowledge, and (2) language features are more discriminable than visual features, and there are large semantic gaps between the features of the two modalities, especially in the lower layers. Therefore, we only incorporate MMA to a few higher lay-ers of transformers to achieve an optimal balance between discrimination and generalization. We evaluate the effectiveness of our approach on three tasks: generalization to novel classes, novel target datasets, and domain generalization. Compared to many state-of-the-art methods, our MMA achieves leading performance in all evaluations. Code is at https://github.com/ZjjConan/Multi-Modal-Adapter
BibTeX
@inproceedings{yang2024mma,
  author = {Lingxiao Yang and Ru-Yuan Zhang and Yanchen Wang and Xiaohua Xie},
  title = {{MMA: Multi-Modal Adapter for Vision-Language Models}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2024},
  doi = {10.1109/cvpr52733.2024.02249},
}
Figure