Resolving Parameter Interference in Model Merging

You are reading version 1 of this direction, frozen 2026-10-04. The latest version is version 2.

Version 1 of 2768 papers1,083 conceptsfrozen 2026-10-04

Model merging has emerged as a "free lunch" in deep learning: a way to combine specialized experts—like a coding model and a math model—into a single multitask model without the astronomical cost of retraining. However, simply averaging the weights of two models often leads to a "performance collapse" where the resulting model is worse than either of its parents.

The core challenge is parameter interference. When we merge models, we aren't just averaging numbers; we are attempting to reconcile divergent signals in a high-dimensional landscape.

The Intuition of Destructive Interference

To understand why merging is difficult, consider a single parameter in a neural network. Suppose a base model is fine-tuned on two different tasks:

  1. Task A (Coding): To excel, it learns that a specific weight should increase (+0.5).
  2. Task B (Math): To excel, it learns that the same weight should decrease (-0.5).

If we use simple weight averaging, these updates cancel each other out, resulting in a change of zero. The merged model loses the specialized knowledge of both tasks. Mohit Bansal, Prateek Yadav, and their colleagues identified this as sign disagreement in their work on TIES-Merging. They found that as more models are added, the probability of these conflicting updates increases, leading to a rapid decline in performance.

Trimming the Noise: Less is More

A surprising discovery in this field is that most fine-tuning updates are actually redundant or represent "noise" relative to the merge. Le Yu and Bowen Yu demonstrated this with DARE (Drop And REscale), showing that one can drop 90% to 99% of the weight updates from a fine-tuned model and still retain its performance Language Models are Super Mario.

By sparsifying the models before merging, we reduce the "surface area" for interference. If Task A only modifies 10% of the weights and Task B modifies a different 10%, the chance of them clashing on the same parameter drops significantly. This concept is also central to TIES-Merging, which trims small updates, elects a dominant sign for each parameter based on magnitude, and only then merges the values.

Aligning Disparate Parameter Spaces

Sometimes, models cannot be merged because they have drifted too far apart or were trained independently without a shared starting point. Hanyang Peng and Qi Wang describe this as "dissonance" in Symphony-MoE. If two models occupy different regions of the parameter space, averaging them is like trying to blend two languages by averaging their alphabets—the result is gibberish.

To fix this, researchers use alignment techniques:

  • Permutation Alignment: Bowen Tian and Songning Lai developed AutoFusion, which uses a Sinkhorn operator to learn how to permute the neurons of one model to match another before merging Wolf2Pack.
  • Geometric Alignment: Zhongyong Chen and Sen Cui proposed EBM-CoT, which treats task updates as directions in high-dimensional space and uses Equiangular Tight Frames (ETF) to ensure these directions don't clash From Coefficients to Directions.

Beyond Simple Averages: Adaptive and Second-Order Merging

If simple averaging is too blunt, how do we decide which model "wins" for a specific parameter?

One approach is uncertainty-based merging. Nico Daheim and Mohammad Emtiyaz Khan treat merging as a Bayesian problem, using the Hessian matrix (which measures the "flatness" or uncertainty of the loss landscape) to weight the merge Model Merging by Uncertainty-Based Gradient Matching. If a model is very "certain" about a weight change, it gets more influence in the final average.

Alternatively, AdaMerging, proposed by Enneng Yang and Li Shen, learns the merging coefficients automatically by minimizing entropy on a small set of unlabeled data AdaMerging: Adaptive Model Merging for Multi-Task Learning. This allows the model to dynamically balance its skills based on the actual distribution of tasks it encounters.

The Scalability Ceiling

While these methods significantly improve performance, a recurring theme in the literature is the "interference ceiling." As noted in Merging Continual Pretraining Models by Kentaro Ueda, merging two experts often works well, but adding a third or fourth model often leads to diminishing returns or performance drops.

This suggests that while we can mitigate interference through clever pruning and alignment, there is a fundamental limit to how much distinct knowledge can be compressed into a single set of weights. This has led to hybrid approaches like Channel Merging and Symphony-MoE, which transition from pure weight merging toward more modular, Mixture-of-Experts (MoE) architectures to preserve specialization.

Papers in this version

Showing 60 of 768 papers.

History & citationVersion 2 · 2026-10-04

Current · Permanent link

  1. Version 2 (latest) · 2026-10-04 · 550 papers
  2. Version 1 · 2026-10-04 · 768 papers
Cite this direction

Latest version (a living page):

@misc{lacuna_parameter_interference_86919955,
  title = {parameter interference},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/parameter-interference/rdr_869199558d8b4d94a6f99e1f64a2ba95},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-19},
}

This exact version (version 1, frozen text):

@misc{lacuna_resolving_parameter_interference_in_mode_86919955_v1,
  title = {Resolving Parameter Interference in Model Merging},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/resolving-parameter-interference-in-model-merging/rdr_869199558d8b4d94a6f99e1f64a2ba95/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-19},
}