Introduction
REMEDY (REcipe MErging DYnamics) is a novel framework designed to merge task-specific Large Vision-Language Models (LVLMs) into a single, versatile powerhouse. By identifying modular "recipes"—specifically the visual projector and the first few layers of the Large Language Model (LLM)—and using a modality-aware allocator to dynamically weight them based on the input, REMEDY significantly boosts performance on both multi-task learning and zero-shot generalization.
The Challenge of Merging LVLMs
Model merging has become a popular strategy to combine specialized models (e.g., one for object detection, another for captioning) without the massive computational cost of retraining from scratch. While this approach works well for unimodal vision models, LVLMs like LLaVA present two major hurdles:
- Scale and Complexity: LVLMs are massive (often 7B+ parameters) and composed of distinct architectural parts, including a vision encoder, a projector, and an LLM. Merging all parameters is computationally heavy and frequently leads to "knowledge interference," where optimizing for one task degrades the weights needed for another.
- Multi-modal Discrepancies: Unlike pure vision models, an LVLM’s behavior depends on both an image and a text prompt. The same image might require a short caption in one task but a detailed scientific explanation in another. Static merging methods that rely on fixed weights cannot adapt to these shifting, multi-modal requirements.
Methodology: The REMEDY Framework
To address these challenges, the authors introduce a scalable, multi-stage paradigm called REMEDY.
1. Recipe Construction
Instead of attempting to merge the entire model, the authors isolate the most "reusable" components, termed recipes. Through extensive ablation studies, they discovered that fine-tuning just the visual projector and the first two layers of the LLM (using LoRA) is often more effective than fine-tuning the entire model. This specific configuration enhances visual-language alignment without overwriting the deep, pre-trained reasoning capabilities of the LLM.
2. Modality-Aware Allocator
To handle the dynamic fusion of these recipes, REMEDY employs a "Modality-aware Allocator." * Token Understanding: It utilizes a self-attention mechanism to analyze the incoming sequence of both image and text tokens. * One-Shot Weighting: Unlike standard Mixture-of-Experts (MoE) models that route every single token differently (which introduces significant latency), REMEDY generates a single set of fusion weights for the entire sequence in one go. * Dynamic Fusion: The allocator dynamically adjusts weights based on the input context. If the input resembles a captioning task, it assigns higher weight to the "captioning recipe." If it’s a scientific query, it leans on the "ScienceQA recipe."
Datasets and Evaluation
The researchers evaluated REMEDY across a broad spectrum of multi-modal scenarios:
- Seen Tasks (Training Recipes): Evaluated on Flickr30k, COCO (Image Captioning), ScienceQA (image-based), and TextVQA.
- Unseen Tasks (Zero-Shot Generalization): Tested on MM-Vet, MM-Bench, MM-Bench-Chinese, VizWiz (VQA for the visually impaired), and POPE (evaluating hallucinations).
- Related Task Analysis: The team used TextCaps, which shares images with TextVQA but requires different instructions, to test how well the model distinguishes between similar visual inputs that have different linguistic goals.
Key Insights: The "Shallow" Advantage
- The "Shallow" Advantage: The paper demonstrates that the initial layers of the LLM are the most critical for adapting to new multi-modal tasks. Fine-tuning deeper layers can actually lead to overfitting and a loss of general, pre-trained knowledge.
- Projectors are Non-Negotiable: Skipping the projector during fine-tuning leads to severe "visual hallucinations." As shown in the qualitative examples, a model fine-tuned without the projector might "see" a flag in an image simply because flags are common in the training data for that region, even if one is not actually present in the prompt image.
- Global vs. Local Routing: By generating weights at the sequence level rather than the token level, REMEDY remains computationally efficient while capturing the global context of the user's request better than traditional MoE approaches.
Key Results
REMEDY consistently outperformed standard merging techniques, including Task Arithmetic, TIES-Merging, and AdaMerging.
- Quantitative Superiority: On the H-score (a harmonic mean measuring both seen and unseen task performance), REMEDY scored 69.25, significantly higher than the Zero-shot baseline (66.31) and Simple Averaging (67.31).
- Zero-Shot Prowess: It was the only method that consistently outperformed the zero-shot baseline across all five unseen datasets, proving it doesn't just "memorize" the training tasks but actually learns to generalize cross-modal knowledge.
- Qualitative Improvements: In head-to-head comparisons, REMEDY produced more detailed and accurate descriptions. For instance, while simple averaging might miss small text in an image, REMEDY’s dynamic allocation allows it to "focus" on the TextVQA recipe to read fine print accurately.
Limitations and Future Work
While REMEDY represents a major step forward for LVLM fusion, the authors note a few areas for future growth:
- Recipe Selection: The "first two layers" rule was found experimentally for LLaVA-1.5. Different architectures might require different "recipe" definitions.
- Dynamic Recognition: Future work could involve more sophisticated, automated ways to identify which modules should become recipes without relying on manual ablation.
- Scaling: The authors aim to test REMEDY on even larger-scale multimodal sets to see if the "one-shot" allocation holds up as the number of specialized tasks scales into the hundreds.
Summary updated