An Empirical Study of Multimodal Model Merging

Yi-Lin Sung, Linjie Li, Kevin Lin, Zhe Gan, Mohit Bansal, Lijuan Wang

Abstract

Model merging (e.g., via interpolation or task arithmetic) fuses multiple models trained on different tasks to generate a multi-task solution. The technique has been proven successful in previous studies, where the models are trained on similar tasks and with the same initialization. In this paper, we expand on this concept to a multimodal setup by merging transformers trained on different modalities. Furthermore, we conduct our study for a novel goal where we can merge vision, language, and cross-modal transformers of a modality-specific architecture to create a parameter-efficient modality-agnostic architecture. Through comprehensive experiments, we systematically investigate the key factors impacting model performance after merging, including initialization, merging mechanisms, and model architectures. Our analysis leads to an effective training recipe for matching the performance of the modality-agnostic baseline via model merging. Our method also outperforms naive merging significantly on various tasks, with improvements of 3% on VQA, 7% on COCO retrieval, 25% on NLVR2, 14% on Flickr30k and 3% on ADE20k.

Introduction

In Vision-Language (VL) modeling, researchers often face a trade-off between two architectural paradigms. Modality-specific models utilize dedicated transformer encoders for images and text, offering high performance at the cost of high parameter counts. Conversely, modality-agnostic models employ a single shared transformer for all inputs, which is parameter-efficient but computationally expensive to train from scratch.

This study explores "multimodal model merging"—the fusion of specialized vision and language transformers into a single, modality-agnostic model. While model merging has traditionally been applied to models trained on similar tasks with identical initializations, this work addresses the challenge of merging models trained on disparate modalities. The primary obstacle is weight drift; weights trained on different data typically migrate into different "basins" in the loss landscape, rendering simple averaging ineffective.

Methodology

Overview of the multimodal model merging framework, illustrating seed pre-training, modality-specific specialization, and merging methods.

The researchers investigated three primary merging mechanisms to unify modality-specific weights:

  1. Interpolation: A weighted average of weights ($\alpha W^v + (1-\alpha)W^l$).
  2. Modality Arithmetic: Treating the delta between a fine-tuned model and its initialization as a "modality vector" and summing these vectors.
  3. RegMean: A closed-form solution designed to minimize the distance of features before and after merging.

The Role of Seed Pre-training

A critical discovery in this work is the necessity of Seed Pre-training. Rather than starting from divergent initializations, the authors utilize a "seed" model that undergoes a brief training phase on both modalities. This ensures that when the vision and language branches are subsequently specialized, they remain within the same optimization basin, maintaining compatibility for later merging.

Experimental Setup

Models were pre-trained on a standard 10M image-text corpus, including COCO, Visual Genome, SBU Captions, and Conceptual Captions. Evaluation spanned a diverse set of benchmarks:

  • Vision-Language Tasks: VQA, NLVR², and Image-Text Retrieval (COCO and Flickr30k).
  • Unimodal Vision Tasks: ImageNet-1k (classification) and ADE20k (segmentation) to verify that the merged model retains unimodal capabilities.

Results and Analysis

Performance on VQA and COCO retrieval as a function of seed pre-training iterations, demonstrating how seed training aligns weights for effective merging.

The proposed merging recipe significantly outperformed naive merging (merging without seed pre-training). Key findings include:

  • Performance Gains: The method achieved a +3% improvement on VQA, +7% on COCO Retrieval, and a substantial +25% on NLVR² compared to naive baselines.
  • Seed Iterations: As shown in the experimental data, increasing seed pre-training iterations helps align the weights, though even a small amount of seed training provides a massive boost over zero-seed baselines.
  • Weight Bias: Optimal results were frequently achieved with an interpolation ratio of 0.75 for vision and 0.25 for language, suggesting visual features may be more critical or harder to preserve during the merge.
  • Predictive Metrics: The authors introduced Truncated Soft Sign Dissimilarity (TSSD), a metric that analyzes weight sign disagreements to predict post-merge performance drops before evaluation.

Conclusion

This research demonstrates that specialized encoders can be effectively "recycled" into general-purpose modality-agnostic models without the full cost of retraining from scratch. By utilizing seed pre-training to maintain weight compatibility, multimodal model merging offers a path toward more modular and parameter-efficient AI development. Future work will explore the applicability of this method to more distant modalities, such as audio, and the merging of models already fine-tuned for specific downstream domains.

Summary updated

Abstract

Model merging (e.g., via interpolation or task arithmetic) fuses multiple models trained on different tasks to generate a multi-task solution. The technique has been proven successful in previous studies, where the models are trained on similar tasks and with the same initialization. In this paper, we expand on this concept to a multimodal setup by merging transformers trained on different modalities. Furthermore, we conduct our study for a novel goal where we can merge vision, language, and cross-modal transformers of a modality-specific architecture to create a parameter-efficient modality-agnostic architecture. Through comprehensive experiments, we systematically investigate the key factors impacting model performance after merging, including initialization, merging mechanisms, and model architectures. We also propose two metrics that assess the distance between weights to be merged and can serve as an indicator of the merging outcomes. Our analysis leads to an effective training recipe for matching the performance of the modality-agnostic baseline (i.e., pre-trained from scratch) via model merging. Our method also outperforms naive merging significantly on various tasks, with improvements of 3% on VQA, 7% on COCO retrieval, 25% on NLVR2, 14% on Flickr30k and 3% on ADE20k. Our code is available at https://github.com/ylsung/vl-merging
BibTeX
@inproceedings{sung2023empirical,
  author = {Yi-Lin Sung and Linjie Li and Kevin Lin and Zhe Gan and Mohit Bansal and Lijuan Wang},
  title = {{An Empirical Study of Multimodal Model Merging}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2023},
  url = {https://openreview.net/forum?id=vkW4GhT7hN},
}
Figure