REMEDY: Recipe Merging Dynamics in Large Vision-Language Models

Didi Zhu, Yibing Song, Tao Shen, Ziyu Zhao, Jinluan Yang, Min Zhang, Chao Wu

Introduction

REMEDY (REcipe MErging DYnamics) is a novel framework designed to merge task-specific Large Vision-Language Models (LVLMs) into a single, versatile powerhouse. By identifying modular "recipes"—specifically the visual projector and the first few layers of the Large Language Model (LLM)—and using a modality-aware allocator to dynamically weight them based on the input, REMEDY significantly boosts performance on both multi-task learning and zero-shot generalization.

The Challenge of Merging LVLMs

Model merging has become a popular strategy to combine specialized models (e.g., one for object detection, another for captioning) without the massive computational cost of retraining from scratch. While this approach works well for unimodal vision models, LVLMs like LLaVA present two major hurdles:

  1. Scale and Complexity: LVLMs are massive (often 7B+ parameters) and composed of distinct architectural parts, including a vision encoder, a projector, and an LLM. Merging all parameters is computationally heavy and frequently leads to "knowledge interference," where optimizing for one task degrades the weights needed for another.
  2. Multi-modal Discrepancies: Unlike pure vision models, an LVLM’s behavior depends on both an image and a text prompt. The same image might require a short caption in one task but a detailed scientific explanation in another. Static merging methods that rely on fixed weights cannot adapt to these shifting, multi-modal requirements.

Methodology: The REMEDY Framework

To address these challenges, the authors introduce a scalable, multi-stage paradigm called REMEDY.

REMEDY Architecture and Stages

1. Recipe Construction

Instead of attempting to merge the entire model, the authors isolate the most "reusable" components, termed recipes. Through extensive ablation studies, they discovered that fine-tuning just the visual projector and the first two layers of the LLM (using LoRA) is often more effective than fine-tuning the entire model. This specific configuration enhances visual-language alignment without overwriting the deep, pre-trained reasoning capabilities of the LLM.

2. Modality-Aware Allocator

To handle the dynamic fusion of these recipes, REMEDY employs a "Modality-aware Allocator." * Token Understanding: It utilizes a self-attention mechanism to analyze the incoming sequence of both image and text tokens. * One-Shot Weighting: Unlike standard Mixture-of-Experts (MoE) models that route every single token differently (which introduces significant latency), REMEDY generates a single set of fusion weights for the entire sequence in one go. * Dynamic Fusion: The allocator dynamically adjusts weights based on the input context. If the input resembles a captioning task, it assigns higher weight to the "captioning recipe." If it’s a scientific query, it leans on the "ScienceQA recipe."

Datasets and Evaluation

The researchers evaluated REMEDY across a broad spectrum of multi-modal scenarios:

  • Seen Tasks (Training Recipes): Evaluated on Flickr30k, COCO (Image Captioning), ScienceQA (image-based), and TextVQA.
  • Unseen Tasks (Zero-Shot Generalization): Tested on MM-Vet, MM-Bench, MM-Bench-Chinese, VizWiz (VQA for the visually impaired), and POPE (evaluating hallucinations).
  • Related Task Analysis: The team used TextCaps, which shares images with TextVQA but requires different instructions, to test how well the model distinguishes between similar visual inputs that have different linguistic goals.

Key Insights: The "Shallow" Advantage

Impact of Shallow Layers and Projector Fine-tuning
  • The "Shallow" Advantage: The paper demonstrates that the initial layers of the LLM are the most critical for adapting to new multi-modal tasks. Fine-tuning deeper layers can actually lead to overfitting and a loss of general, pre-trained knowledge.
  • Projectors are Non-Negotiable: Skipping the projector during fine-tuning leads to severe "visual hallucinations." As shown in the qualitative examples, a model fine-tuned without the projector might "see" a flag in an image simply because flags are common in the training data for that region, even if one is not actually present in the prompt image.
  • Global vs. Local Routing: By generating weights at the sequence level rather than the token level, REMEDY remains computationally efficient while capturing the global context of the user's request better than traditional MoE approaches.

Key Results

REMEDY consistently outperformed standard merging techniques, including Task Arithmetic, TIES-Merging, and AdaMerging.

  • Quantitative Superiority: On the H-score (a harmonic mean measuring both seen and unseen task performance), REMEDY scored 69.25, significantly higher than the Zero-shot baseline (66.31) and Simple Averaging (67.31).
  • Zero-Shot Prowess: It was the only method that consistently outperformed the zero-shot baseline across all five unseen datasets, proving it doesn't just "memorize" the training tasks but actually learns to generalize cross-modal knowledge.
  • Qualitative Improvements: In head-to-head comparisons, REMEDY produced more detailed and accurate descriptions. For instance, while simple averaging might miss small text in an image, REMEDY’s dynamic allocation allows it to "focus" on the TextVQA recipe to read fine print accurately.

Limitations and Future Work

While REMEDY represents a major step forward for LVLM fusion, the authors note a few areas for future growth:

  • Recipe Selection: The "first two layers" rule was found experimentally for LLaVA-1.5. Different architectures might require different "recipe" definitions.
  • Dynamic Recognition: Future work could involve more sophisticated, automated ways to identify which modules should become recipes without relying on manual ablation.
  • Scaling: The authors aim to test REMEDY on even larger-scale multimodal sets to see if the "one-shot" allocation holds up as the number of specialized tasks scales into the hundreds.

Summary updated

Abstract

Model merging has emerged as a powerful technique for combining task-specific vision models into a unified and multi-functional model. Previous methods represented by task arithmetic, have demonstrated effectiveness and scalability in this domain. When large vision-language models (LVLMs) arise with model size scaling up, this design becomes challenging to fuse different instruction-tuned LVLMs for generalization enhancement. The large scale and multi-modal nature of LVLMs present unique obstacles, including constructing reusable and modular components to accommodate the multi-component architecture of LVLMs and the requirement for dynamic fusion based on multi-modal input tokens. To address these challenges, we propose the \textbf{RE}cipe \textbf{ME}rging \textbf{DY}namics (REMEDY) method, a scalable and flexible paradigm for model merging in LVLMs. We first define reusable modules termed \textit{recipes} including the projector and shallow LLM layers, enhancing visual-language understanding. Then, we introduce a modality-aware allocator dynamically generates weights in a one-shot manner based on input relevance to existing recipes, enabling efficient cross-modal knowledge integration. REMEDY thus offers an adaptive solution for LVLMs to tackle both seen (i.e., multi-task learning) and unseen (i.e., zero-shot generalization) tasks. Experimental results demonstrate that our method consistently improves performance on both seen and unseen tasks, underscoring the effectiveness of REMEDY in diverse multi-modal scenarios.
BibTeX
@inproceedings{zhu2025remedy,
  author = {Didi Zhu and Yibing Song and Tao Shen and Ziyu Zhao and Jinluan Yang and Min Zhang and Chao Wu},
  title = {{REMEDY: Recipe Merging Dynamics in Large Vision-Language Models}},
  booktitle = {International Conference on Learning Representations},
  year = {2025},
  url = {https://openreview.net/forum?id=iX7eHHE5Tx},
}
Figure