One way to get a single multimodal model with fewer parameters is to train one transformer briefly on image–text pairs, copy it into a vision branch and a language branch, specialize each copy on its own modality, and then average the copies back into one shared network. An Empirical Study of Multimodal Model Merging shows this works only when the copies start from the same short "seed" training run. Its diagnostic, Truncated Soft Sign Dissimilarity (TSSD), measures how often the two branches pushed the same weights in opposite directions, and high values predict a broken merge. Yet the study stops at the averaging step. Nobody then trains the merged model to repair the damage, and nobody uses the conflict measurement to decide where repair is needed.
We propose to treat the merge as the midpoint of a staged curriculum and to use the conflict map as an allocation rule. After merging, we measure sign conflict layer by layer. We attach small trainable low-rank corrections, one per modality, only to the most conflicted layers. Every other weight stays frozen and shared, and we run a short joint fine-tune. The bet is that a fixed, small budget of trainable parameters recovers more accuracy when placed by measured conflict than when spread uniformly or placed by hand. If it does, the same map should also tell us how much seed training a merge actually needs.
Core Claim
Sign conflict between specialized branches is a usable signal for where a shared multimodal backbone needs modality-specific capacity. Concretely, conflict-guided, modality-routed low-rank adapters on the merged model should beat four alternatives at the same trainable-parameter and compute budget: uniformly placed adapters, the same adapters placed in the least-conflicted layers, and continuing to train the seed model jointly without branching and merging.
The proposal differs from three bodies of work. Learning Visual Representation from Modality-Shared Contrastive Language-Image Pre-training (MS-CLIP) shares all attention and feed-forward weights across vision and text but chooses by hand which parts stay modality-specific: per-modality LayerNorm and an early specialized block. We derive that choice from measured disagreement after merging, rather than fixing it before training. UniMRSeg (UniMRSeg) runs a three-stage curriculum on a fully shared segmentation network but ends with a frozen backbone and one adapter in a fixed location. Its authors flag simplifying the stages and parameter-efficient fine-tuning as future work. Multimodal Lego shows that brief fine-tuning after a zero-shot merge (LegoFuse, about 2 epochs) beats the merge alone, which motivates a repair stage. However, it works in a Fourier-transformed latent space on biomedical data and does not localize where tuning is needed.
Research Program
The pipeline has four stages and follows the merging study's protocol at reduced scale.
In the first stage, a ViT-style transformer is seed-trained on image–text pairs. The second stage branches it into a vision-specialized copy and a language-specialized copy, each trained on unimodal data.
The third stage merges the two copies. Following An Empirical Study of Multimodal Model Merging, we use linear interpolation with its reported 0.75 weighting toward vision. RegMean is a secondary merge operator. At this stage we compute, for every weight matrix $l$, a conflict score $c_l$, defined as the TSSD between the two branches' updates relative to the seed weights.
The fourth stage is harmonization. We rank layers by $c_l$ and give the top-ranked layers low-rank corrections, adding to frozen merged weights in the same style as STEP's use of ReLoRA (STEP). Each correction comes as a vision copy and a language copy, selected by the input token's modality, and we keep MS-CLIP-style per-modality LayerNorm. Layers are added until a fixed trainable-parameter budget $B$ is used up. The model is then jointly fine-tuned on mixed image–text data for a short, fixed number of steps.
Three analyses come out of this pipeline. First, the curve of downstream accuracy against $B$ for conflict-guided versus alternative placements. Second, whether $c_l$ measured right after merging predicts which layers' corrections grow largest during harmonization, as a mechanistic check on the signal. Third, a sweep over seed-training length. The merging study reports that naive merges of unrelated models collapse. We ask whether conflict-guided harmonization lets a much shorter seed run reach the same final quality, which would cut the dominant cost of the method.
Baselines
The naive reference point is the merged model with no harmonization: interpolation or RegMean exactly as in An Empirical Study of Multimodal Model Merging. That paper reports the merged model reaching parity with a modality-agnostic model trained from scratch. Relative to naive merging, it reports gains of +3% on VQAv2, +7% on COCO retrieval and +25% on NLVR2. No absolute post-merge accuracies were found on Lacuna for these benchmarks, so we will reproduce this baseline ourselves at our scale rather than quote a target.
The strongest competitor is the compute-matched continuation of the seed model. Here the seed model is trained jointly on the same mixed image–text data for the combined step count of branching plus harmonization, with no branching or merging. If our pipeline cannot beat this, merging is an expensive detour. Data, step counts, peak memory, and the hyperparameter search budget (the same number of learning-rate and schedule trials) will be matched.
The second strong competitor is full joint fine-tuning of the merged model. This is the obvious Stage 3 "global harmonization" from the staged-training literature. It gets the same step count and tuning budget. It bounds what our partial unfreezing could hope to reach, and it tests whether unfreezing everything causes the merged model to lose unimodal ability.
The key ablation keeps the budget $B$, adapter type and training schedule fixed, and varies only placement. Placements are top-conflict layers, uniform across all layers, bottom-conflict layers, and random layers. A second ablation replaces the per-modality corrections with a single shared correction, to test whether routing by modality matters or extra capacity alone does.
Beating the baselines means two things. Conflict-guided placement must exceed uniform placement at equal $B$ across at least three seeds on VQAv2, NLVR2 and COCO retrieval. It must also match or exceed compute-matched seed continuation while training under a few percent of the backbone's parameters in the final stage. For external context, Multi-stage Pre-training over Simplified Multimodal Pre-training Models reports 74.72% on NLVR2 with a 84.3M-parameter cross-modal model. That result uses a different architecture and data scale, so it is a sanity anchor rather than a target.
Why not just…?
Why not just keep training the seed model jointly and skip merging? This is the real threat. The merging study frames merging as a route to modality-agnostic models, which it describes as notoriously hard to train from scratch. But at matched compute it does not settle whether branch-then-merge beats simple continued joint training. PolyViT (PolyViT) suggests mixed-modality co-training is workable only with careful task sampling, which favors branching. That is suggestive, not decisive. Our compute-matched baseline answers the question directly. If continuation wins, we will report that the conflict map is still useful as a diagnostic, and drop the claim that merging pays off.
Why not just fully fine-tune the merged model? It may match our accuracy at high data. Modular and Parameter-Efficient Multimodal Fusion with Prompting shows tiny trainable budgets hit a ceiling at high resource, so we expect full fine-tuning to win at the largest budget. Our claim concerns the accuracy-per-trainable-parameter curve and retention of unimodal ability. We will measure retention with ImageNet linear probing as a vision check, alongside the language branch's own pretraining loss. If full fine-tuning dominates at every budget and forgets nothing, the proposal loses its reason to exist.
Why not just put LoRA everywhere, or hand-pick modality-specific parts like MS-CLIP? Uniform placement and hand-picked placement are exactly the ablation arms. If uniform placement ties conflict-guided placement, the conflict signal carries no information about where capacity belongs, and the method reduces to standard parameter-efficient tuning. We would report that as a negative result.
Proposed Experiments
All runs use the four-stage protocol at a scale where the seed-length sweep is affordable. That means a seed run well below the merging study's 10M image–text pairs, with seed lengths spanning several factors of two down to near zero. Evaluation covers VQAv2 accuracy, NLVR2 accuracy, COCO and Flickr30k retrieval recall@1, and ImageNet zero-shot and linear-probe accuracy for vision retention. We also record trainable parameters, harmonization steps and peak memory.
Three checks decide the outcome. The first is whether top-conflict placement beats uniform and bottom-conflict placement at equal budget, with gaps larger than seed-to-seed variance. The second is whether the whole pipeline matches compute-matched seed continuation. The third is whether, as seed length shrinks, conflict-guided harmonization degrades more gracefully than uniform adapters or no harmonization, letting a shorter seed reach the same downstream quality.
A secondary check tests whether the method's logic transfers beyond vision–language. We apply the same conflict-guided placement to a fully shared segmentation backbone on DELIVER, using MAGIC (Centering the Value of Every Modality), where sensor branches are an equally natural unit. MAGIC reports a 60% parameter reduction and a 19.41% average mIoU gain under sensor failure, against which robustness can be compared.
References
- An Empirical Study of Multimodal Model Merging
- Learning Visual Representation from Modality-Shared Contrastive Language-Image Pre-training
- UniMRSeg: Unified Modality-Relax Segmentation via Hierarchical Self-Supervised Compensation
- Multimodal Lego: Model Merging and Fine-Tuning Across Topologies and Modalities
- STEP: Staged Parameter-Efficient Pre-training for Large Language Models
- Multi-stage Pre-training over Simplified Multimodal Pre-training Models
- PolyViT: Co-training Vision Transformers on Images, Videos and Audio
- Modular and Parameter-Efficient Multimodal Fusion with Prompting
- Centering the Value of Every Modality: Towards Efficient and Resilient Modality-Agnostic Semantic Segmentation