TL;DR
Model merging—combining specialized models by adding their "task vectors"—often fails due to task interference that is only detectable after the merge is built. This paper demonstrates that interference can be predicted beforehand using first-order derivatives, but only if they are calculated at the fine-tuned "receiver" model rather than the pretrained starting point. This insight leads to LR-Merge, a method that optimizes merge coefficients 1000x faster than traditional search by using a cheap, linear surrogate model.
What Problem Does This Paper Address?
In the world of modular AI, "model merging" is a popular way to create multi-task models without expensive joint training. You take a pretrained backbone and several specialized versions of it (fine-tuned for different tasks), calculate their "task vectors" (the difference between fine-tuned and pretrained weights), and add them together.
The catch? Interference. Adding Task B's vector might drastically degrade Task A's performance. To fix this, researchers tune "merge coefficients" (weights for each task vector), but finding the right ones currently requires a "trial and error" loop: build a candidate model, evaluate it on data, adjust coefficients, and repeat. This is computationally expensive and slow.
Methods
The authors propose using a first-order quantity—the Jacobian-vector product (JVP)—to forecast how a task's outputs will change when another task's vector is added.
The "Where" Matters
The critical discovery is the linearization point. Standard theory often suggests linearizing at the pretrained initialization ($\theta_0$). However, the authors show that for vision models, a forecast made at $\theta_0$ is nearly useless (often pointing in a direction orthogonal to the actual change). Instead, you must linearize at the fine-tuned model ($\theta_A$) that is receiving the update.
LR-Merge (Linear-Response Merge)
Building on this, they introduce LR-Merge:
- Freeze Responses: Compute the JVPs at the current merge point and freeze them.
- Affine Surrogate: This creates a "surrogate" model where the outputs are simple linear (affine) combinations of the coefficients.
- Convex Optimization: Finding the best coefficients becomes a convex "teacher matching" problem. You optimize the coefficients to make the surrogate's outputs match the original specialized "teacher" models on a small set of unlabeled probes.
- No Backprop: This optimization happens entirely on the surrogate, meaning you never have to build or backpropagate through the full merged network during the search.
Datasets
The researchers evaluated their approach across both vision and language domains:
- Vision: CLIP backbones (ViT-B/32, B/16, L/14) tested on suites of 6, 8, 14, and 20 tasks (including MNIST, SVHN, GTSRB, EuroSAT, DTD, and CIFAR-10).
- Language: RoBERTa-base and Pythia (410M and 1.4B parameters) using LoRA fine-tuning on GLUE tasks and other language benchmarks.
Key Insights and Ideas
- The "Early Re-aiming" Phenomenon: Why does linearizing at the pretrained base fail? The authors found that within the first few dozen steps of fine-tuning, the model "re-aims" its cross-task responses. Even though the weights haven't moved far in terms of distance, the direction of the Jacobian has shifted significantly.
- Direction Over Distance: The failure of the pretrained anchor isn't about how far the model has moved from the start, but the direction of that movement along the receiver's own task vector.
- Surrogate Efficiency: Once the surrogate is built, scoring a new set of coefficients is roughly 1000x faster than evaluating the actual merged model. This allows for much more granular and complex coefficient tuning (e.g., per-layer or per-block coefficients).
Key Results
- Universal Improvement: LR-Merge improved every base merger it was applied to (including RegMean, Iso-C, and TIES). On an 8-task vision benchmark, it boosted RegMean accuracy by over 2 percentage points.
- Scaling with Complexity: The performance gains from LR-Merge generally increased as the number of merged tasks grew, proving particularly effective for large-scale merges (up to 20 tasks).
- Cost-Effectiveness: LR-Merge matches the accuracy of "exact teacher matching" (which requires full network passes) but converges in about one-third of the time.
- Label-Free: The method works using only unlabeled "probes," making it highly practical for scenarios where original training data is unavailable.
Limitations and Future Work
- Locality Limits: Because it relies on first-order derivatives, a single "read" is only accurate for nearby merges. For a complete multi-task merge where coefficients are large, the authors found that two "reads" (linearizing at two points along the merge path) were necessary to maintain high accuracy.
- Generative Tasks: The current work focuses on logit-based tasks (classification). The authors note that while the forecast ranks loss changes in language models, they make no claims about its effectiveness for free-form text generation.
- Architecture Requirements: The method assumes a shared initialization and architecture across all models being merged.
Summary updated