Watch Where You Linearize: Predicting and Correcting Model-Merging Interference

Ido Nitzan Hidekel, Dan RavivarXiv 2026 · arXiv

TL;DR

Model merging—combining specialized models by adding their "task vectors"—often fails due to task interference that is only detectable after the merge is built. This paper demonstrates that interference can be predicted beforehand using first-order derivatives, but only if they are calculated at the fine-tuned "receiver" model rather than the pretrained starting point. This insight leads to LR-Merge, a method that optimizes merge coefficients 1000x faster than traditional search by using a cheap, linear surrogate model.

What Problem Does This Paper Address?

In the world of modular AI, "model merging" is a popular way to create multi-task models without expensive joint training. You take a pretrained backbone and several specialized versions of it (fine-tuned for different tasks), calculate their "task vectors" (the difference between fine-tuned and pretrained weights), and add them together.

The catch? Interference. Adding Task B's vector might drastically degrade Task A's performance. To fix this, researchers tune "merge coefficients" (weights for each task vector), but finding the right ones currently requires a "trial and error" loop: build a candidate model, evaluate it on data, adjust coefficients, and repeat. This is computationally expensive and slow.

Methods

The authors propose using a first-order quantity—the Jacobian-vector product (JVP)—to forecast how a task's outputs will change when another task's vector is added.

The "Where" Matters

The critical discovery is the linearization point. Standard theory often suggests linearizing at the pretrained initialization ($\theta_0$). However, the authors show that for vision models, a forecast made at $\theta_0$ is nearly useless (often pointing in a direction orthogonal to the actual change). Instead, you must linearize at the fine-tuned model ($\theta_A$) that is receiving the update.

Figure 1: Interference is receiver-local. zA(θ): logits on task A’s probes; JA=∂zA/∂θ; τB: task B’s vec- tor. Black: realized drift zA(θA+ατB)−zA(θA); blue: forecast at the receiving model, αJA(θA)τB; orange: forecast at the pretrained model, αJA(θ0)τB, which points in a nearly unrelated direction (formal setup in Section 3).

LR-Merge (Linear-Response Merge)

Building on this, they introduce LR-Merge:

  1. Freeze Responses: Compute the JVPs at the current merge point and freeze them.
  2. Affine Surrogate: This creates a "surrogate" model where the outputs are simple linear (affine) combinations of the coefficients.
  3. Convex Optimization: Finding the best coefficients becomes a convex "teacher matching" problem. You optimize the coefficients to make the surrogate's outputs match the original specialized "teacher" models on a small set of unlabeled probes.
  4. No Backprop: This optimization happens entirely on the surrogate, meaning you never have to build or backpropagate through the full merged network during the search.

Datasets

The researchers evaluated their approach across both vision and language domains:

  • Vision: CLIP backbones (ViT-B/32, B/16, L/14) tested on suites of 6, 8, 14, and 20 tasks (including MNIST, SVHN, GTSRB, EuroSAT, DTD, and CIFAR-10).
  • Language: RoBERTa-base and Pythia (410M and 1.4B parameters) using LoRA fine-tuning on GLUE tasks and other language benchmarks.

Key Insights and Ideas

  • The "Early Re-aiming" Phenomenon: Why does linearizing at the pretrained base fail? The authors found that within the first few dozen steps of fine-tuning, the model "re-aims" its cross-task responses. Even though the weights haven't moved far in terms of distance, the direction of the Jacobian has shifted significantly.
  • Direction Over Distance: The failure of the pretrained anchor isn't about how far the model has moved from the start, but the direction of that movement along the receiver's own task vector.
  • Surrogate Efficiency: Once the surrogate is built, scoring a new set of coefficients is roughly 1000x faster than evaluating the actual merged model. This allows for much more granular and complex coefficient tuning (e.g., per-layer or per-block coefficients).
The reorientation of task responses happens very early in the fine-tuning process.

Key Results

  • Universal Improvement: LR-Merge improved every base merger it was applied to (including RegMean, Iso-C, and TIES). On an 8-task vision benchmark, it boosted RegMean accuracy by over 2 percentage points.
  • Scaling with Complexity: The performance gains from LR-Merge generally increased as the number of merged tasks grew, proving particularly effective for large-scale merges (up to 20 tasks).
  • Cost-Effectiveness: LR-Merge matches the accuracy of "exact teacher matching" (which requires full network passes) but converges in about one-third of the time.
  • Label-Free: The method works using only unlabeled "probes," making it highly practical for scenarios where original training data is unavailable.

Limitations and Future Work

  • Locality Limits: Because it relies on first-order derivatives, a single "read" is only accurate for nearby merges. For a complete multi-task merge where coefficients are large, the authors found that two "reads" (linearizing at two points along the merge path) were necessary to maintain high accuracy.
  • Generative Tasks: The current work focuses on logit-based tasks (classification). The authors note that while the forecast ranks loss changes in language models, they make no claims about its effectiveness for free-form text generation.
  • Architecture Requirements: The method assumes a shared initialization and architecture across all models being merged.

Summary updated

Abstract

Model merging combines separately fine-tuned models by adding their task vectors to a shared backbone. Interference is typically assessed only after constructing and evaluating a candidate merge, making coefficient search expensive. We show that a first-order directional derivative of task A's outputs along task B's vector can predict interference, but only when evaluated at the appropriate model. At the receiver model, this derivative forecasts nearby merges per task and per example, halves the accuracy error of a no-change baseline, and identifies most examples whose predictions flip. For full multi-task merges, a single evaluation is insufficient; two evaluations along the merge path are needed. At the shared pretrained backbone, however, the derivative is nearly orthogonal to realized changes in our vision experiments and only partly predictive in LoRA language-model pilots. Controls link this failure to displacement along the receiver's own task vector, not simply parameter distance; the mismatch appears within the first few dozen fine-tuning steps. Freezing receiver-local responses produces a surrogate affine in the merge coefficients. We use it to construct LR-Merge, a label-free correction that optimizes coefficients through convex teacher matching on unlabeled probes, without backpropagation through merged networks. LR-Merge improves all tested closed-form, isotropic, and sparsified base mergers, with gains generally increasing as more tasks are merged. Using pretrained rather than receiver-local responses instead degrades the base merge. LR-Merge matches exact teacher matching through a merged network, which takes roughly three times longer to converge and performs similarly at an equal compute budget. Once built, the surrogate answers new coefficient and task-subset queries in seconds.
BibTeX
@misc{hidekel2026watch,
  author = {Ido Nitzan Hidekel and Dan Raviv},
  title = {{Watch Where You Linearize: Predicting and Correcting Model-Merging Interference}},
  year = {2026},
  eprint = {2611.25000},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.25000},
}
Figure