MAFMO: Multi-modal Adaptive Fusion with Meta-template Optimization for Vision-Language Models

Mingrui Xie, Lulu Xu, Junliang DuEMNLP 2025

TL;DR

This paper introduces MAFMO, a novel framework that significantly enhances the adaptation of large vision-language models (VLMs) to new tasks. By harmonizing cross-modal interactions, dynamically optimizing prompts, and preserving original knowledge, MAFMO achieves state-of-the-art performance on fine-grained recognition tasks with minimal computational overhead.

What Problem Does This Paper Address?

Vision-language models (VLMs) like CLIP have shown remarkable zero-shot generalization capabilities, bridging the gap between visual and textual information. However, adapting these massive models to specific downstream tasks is challenging. Full fine-tuning is computationally prohibitive and prone to overfitting, especially with limited data. Existing parameter-efficient methods often focus on adapting modalities independently, leading to suboptimal cross-modal alignment. Furthermore, VLMs are sensitive to prompt engineering, and naive adaptation can disrupt the delicate alignment learned during pre-training. This paper addresses the need for a robust, parameter-efficient, and effective method to adapt VLMs while preserving their core capabilities and improving cross-modal understanding.

Methods

MAFMO is a plug-and-play framework composed of three key components:

  • Harmonic Cross-Modal Adapter (HCMA): This component introduces a novel "harmonic resonance mechanism" inspired by physics. It selectively enhances aligned cross-modal relationships by using shared transformation layers that interact more strongly with similar natural frequencies. A dynamic calibration mechanism further adjusts the adapter's influence based on the semantic alignment between modalities, ensuring it makes substantial changes only when alignment is poor.
  • Meta-Template Optimization (MTO): To overcome the limitations of fixed prompt templates, MTO dynamically generates and combines diverse prompt structures based on visual content. It constructs a "template space" of learnable prompt tokens and uses an "input-adaptive template selection" mechanism to compute a distribution over these templates based on visual features. This allows for more flexible and context-aware prompting. The framework then fuses these templates using an attention-based co-adaptation mechanism and incorporates diversity regularization to ensure effective utilization of the template space.
  • Cross-Modal Knowledge Synthesis (CMKS): This component ensures that adaptations preserve and enhance the original model's capabilities. It employs three regularization losses:
  • Feature Space Alignment: Enforces that the distributions of adapted embeddings remain similar to original embeddings, preventing drastic shifts.
  • Structure-Preserving Regularization: Maintains the relational structure within each modality by preserving pairwise relationships between features.
  • Cross-Modal Consistency: Ensures consistent cross-modal relationships by preserving the scaled dot-product similarity between visual and textual embeddings.

Datasets

The paper evaluates MAFMO on six fine-grained visual recognition datasets:

  • OxfordPets
  • StanfordCars
  • Food101
  • FGVCAircraft
  • SUN397
  • ImageNetV2

For evaluation, classes are split into 80% base classes for training and 20% novel classes to assess cross-category generalization.

Key Insights and Ideas

  • Harmonic Resonance for Cross-Modal Alignment: The idea of using "harmonic resonance" to strengthen cross-modal interactions is a novel and intuitive approach to improving alignment, especially when modalities are already somewhat aligned.
  • Dynamic Prompting is Crucial: The MTO component highlights that fixed prompts are a bottleneck. Dynamically generating and adapting prompts based on visual input is key to unlocking better generalization.
  • Synergy Over Simple Addition: The MAFMO framework's components don't just add improvements; they synergize. Even though individual components offer benefits, their combination yields superior results, suggesting a holistic design.
  • Robustness Beyond Accuracy: MAFMO not only improves accuracy but also unexpectedly enhances robustness against adversarial perturbations, a critical aspect for real-world applications.
  • Parameter Efficiency Pays Off: The framework consistently improves performance on various baseline models (CLIP, CoOp, Co-CoOp, MaPLe) while maintaining computational tractability, making it a practical enhancement.

Key Results

  • MAFMO consistently improves performance across all evaluated baseline models and datasets, enhancing both base and novel class accuracy.
  • Significant improvements were observed for models with limited cross-modal interaction, such as CoOp+MAFMO showing dramatic gains of 4.54% and 7.53% in novel class accuracy on StanfordCars and FGVCAircraft, respectively.
  • Co-CoOp+MAFMO achieved a 9.48% improvement in harmonic mean accuracy on FGVCAircraft.
  • The framework demonstrates strong performance on challenging datasets with fine-grained distinctions and robustness to distribution shifts (e.g., ImageNetV2).
  • MAFMO significantly enhances few-shot learning capabilities, with improvements of 5.17% in novel class accuracy for CoOp with just one example per class.
  • Cross-dataset experiments show MAFMO consistently improves generalization, substantially narrowing the gap for models like CoOp that struggle with domain shifts.
  • The inference speed analysis shows a modest increase in latency (up to 13% for MaPLe+MAFMO) but maintains high throughput, indicating good computational efficiency.
  • Adversarial robustness analysis shows that MAFMO-enhanced models experience smaller accuracy drops under FGSM attacks compared to their original counterparts.

Limitations and Future Work

The authors acknowledge the following limitations:

  • Modest Computational Overhead: MAFMO introduces a modest computational overhead during training.
  • Architecture Dependency: Performance gains may vary across different base architectures.
  • Domain Specificity: The applicability of MAFMO to specialized domains beyond the evaluated datasets remains unexplored.

Future work suggested includes:

  • Improving computational efficiency.
  • Broadening architectural compatibility.
  • Extending evaluations to more diverse vision-language tasks.

Summary updated

BibTeX
@inproceedings{xie2025mafmo,
  author = {Mingrui Xie and Lulu Xu and Junliang Du},
  title = {{MAFMO: Multi-modal Adaptive Fusion with Meta-template Optimization for Vision-Language Models}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2025},
}
Figure