Cross-Modal Adapter: Parameter-Efficient Transfer Learning Approach for Vision-Language Models

Juncheng Yang, Zuchao Li, Shuai Xie, Weiping Zhu, Wei Yu, Shijun Li

Introduction

In the rapidly evolving landscape of Vision-Language Models (VLMs) like CLIP, the primary challenge has shifted from building larger models to efficiently adapting these foundation models to specific downstream tasks. Traditional full fine-tuning is computationally expensive and often leads to overfitting, especially on small datasets.

While Parameter-Efficient Transfer Learning (PETL) methods like adapters and cache-based systems (e.g., Tip-Adapter) have emerged, they often suffer from two critical limitations: they primarily focus on the image modality, neglecting the structural richness of text, and they treat all training samples as equally easy to learn. The Cross-Modal Adapter (XMAdapter) addresses these gaps by treating both images and text as equally vital components in a searchable knowledge cache and prioritizing "hard" training examples.

The XMAdapter Framework

XMAdapter introduces a dual-modality approach to model adaptation, moving beyond the image-centric focus of previous methods. Its architecture is built upon three core innovations:

  1. Cross-Modal Cache Construction: Unlike standard methods that only cache image features, XMAdapter builds a dual-modality system. It utilizes specialized networks to map both textual and image features into a shared low-dimensional space, allowing the model to retrieve multi-modal clues simultaneously during inference.
  2. Adaptive Scaling: The model dynamically adjusts the fusion ratio between image and text similarities using a learnable coefficient ((\gamma)). This decoupling allows the system to weigh visual evidence against semantic context based on which modality is more reliable for a specific sample.
  3. Hard Example Mining (OHEM): XMAdapter identifies difficult training samples by measuring the "affinity gap"—the difference in similarity scores between the image and text branches. Samples with a large discrepancy are flagged as "hard," and the model adaptively increases learning intensity for these cases to sharpen decision boundaries.
The architecture of XMAdapter, illustrating the dual-modality cache construction and the adaptive fusion of image and text features.

Experimental Results

XMAdapter was evaluated across 11 benchmark datasets, including ImageNet, OxfordPets, and Flowers102, focusing on few-shot classification and domain generalization.

Performance Highlights

  • State-of-the-Art Accuracy: On the 16-shot ImageNet task, XMAdapter achieved 66.22% accuracy, surpassing competitors like Tip-Adapter (65.44%) and GraphAdapter (65.70%).
  • Robust Generalization: When tested on out-of-distribution sets such as ImageNet-Sketch and ImageNet-Adversarial, XMAdapter consistently outperformed other PETL methods across various CLIP backbones (ResNet and ViT).
  • Fine-Grained Recognition: The method showed significant gains on difficult datasets like FGVCAircraft, proving its ability to distinguish subtle differences between highly similar categories.
Average performance comparison across 11 datasets, showing XMAdapter consistently outperforming other state-of-the-art adaptation methods as the number of training examples increases.

Conclusion

XMAdapter demonstrates that smarter data integration and multi-modal retrieval are more effective than simply increasing parameter counts. By leveraging the structural information of text and focusing on hard training examples through affinity-based mining, it provides a highly efficient and robust solution for adapting large-scale Vision-Language Models to specialized tasks. Future work may explore scaling this cross-modal caching approach to generative tasks like image captioning and visual question answering.

Summary updated

BibTeX
@inproceedings{yang2024cross,
  author = {Juncheng Yang and Zuchao Li and Shuai Xie and Weiping Zhu and Wei Yu and Shijun Li},
  title = {{Cross-Modal Adapter: Parameter-Efficient Transfer Learning Approach for Vision-Language Models}},
  booktitle = {ICME},
  year = {2024},
  doi = {10.1109/icme57554.2024.10688369},
}
Figure