Abstract
Model merging (e.g., via interpolation or task arithmetic) fuses multiple models trained on different tasks to generate a multi-task solution. The technique has been proven successful in previous studies, where the models are trained on similar tasks and with the same initialization. In this paper, we expand on this concept to a multimodal setup by merging transformers trained on different modalities. Furthermore, we conduct our study for a novel goal where we can merge vision, language, and cross-modal transformers of a modality-specific architecture to create a parameter-efficient modality-agnostic architecture. Through comprehensive experiments, we systematically investigate the key factors impacting model performance after merging, including initialization, merging mechanisms, and model architectures. Our analysis leads to an effective training recipe for matching the performance of the modality-agnostic baseline via model merging. Our method also outperforms naive merging significantly on various tasks, with improvements of 3% on VQA, 7% on COCO retrieval, 25% on NLVR2, 14% on Flickr30k and 3% on ADE20k.
Introduction
In Vision-Language (VL) modeling, researchers often face a trade-off between two architectural paradigms. Modality-specific models utilize dedicated transformer encoders for images and text, offering high performance at the cost of high parameter counts. Conversely, modality-agnostic models employ a single shared transformer for all inputs, which is parameter-efficient but computationally expensive to train from scratch.
This study explores "multimodal model merging"—the fusion of specialized vision and language transformers into a single, modality-agnostic model. While model merging has traditionally been applied to models trained on similar tasks with identical initializations, this work addresses the challenge of merging models trained on disparate modalities. The primary obstacle is weight drift; weights trained on different data typically migrate into different "basins" in the loss landscape, rendering simple averaging ineffective.
Methodology
The researchers investigated three primary merging mechanisms to unify modality-specific weights:
- Interpolation: A weighted average of weights ($\alpha W^v + (1-\alpha)W^l$).
- Modality Arithmetic: Treating the delta between a fine-tuned model and its initialization as a "modality vector" and summing these vectors.
- RegMean: A closed-form solution designed to minimize the distance of features before and after merging.
The Role of Seed Pre-training
A critical discovery in this work is the necessity of Seed Pre-training. Rather than starting from divergent initializations, the authors utilize a "seed" model that undergoes a brief training phase on both modalities. This ensures that when the vision and language branches are subsequently specialized, they remain within the same optimization basin, maintaining compatibility for later merging.
Experimental Setup
Models were pre-trained on a standard 10M image-text corpus, including COCO, Visual Genome, SBU Captions, and Conceptual Captions. Evaluation spanned a diverse set of benchmarks:
- Vision-Language Tasks: VQA, NLVR², and Image-Text Retrieval (COCO and Flickr30k).
- Unimodal Vision Tasks: ImageNet-1k (classification) and ADE20k (segmentation) to verify that the merged model retains unimodal capabilities.
Results and Analysis
The proposed merging recipe significantly outperformed naive merging (merging without seed pre-training). Key findings include:
- Performance Gains: The method achieved a +3% improvement on VQA, +7% on COCO Retrieval, and a substantial +25% on NLVR² compared to naive baselines.
- Seed Iterations: As shown in the experimental data, increasing seed pre-training iterations helps align the weights, though even a small amount of seed training provides a massive boost over zero-seed baselines.
- Weight Bias: Optimal results were frequently achieved with an interpolation ratio of 0.75 for vision and 0.25 for language, suggesting visual features may be more critical or harder to preserve during the merge.
- Predictive Metrics: The authors introduced Truncated Soft Sign Dissimilarity (TSSD), a metric that analyzes weight sign disagreements to predict post-merge performance drops before evaluation.
Conclusion
This research demonstrates that specialized encoders can be effectively "recycled" into general-purpose modality-agnostic models without the full cost of retraining from scratch. By utilizing seed pre-training to maintain weight compatibility, multimodal model merging offers a path toward more modular and parameter-efficient AI development. Future work will explore the applicability of this method to more distant modalities, such as audio, and the merging of models already fine-tuned for specific downstream domains.
Summary updated