Overcoming Parameter Interference in Model Merging

1,075 concepts575 papers5 proposals

The dream of model merging is simple: take several neural networks, each fine-tuned to be an expert in a specific domain—such as coding, mathematics, or medical diagnosis—and blend their weights together into a single, multi-task powerhouse. This training-free approach promises the benefits of an ensemble without the prohibitive computational and memory costs of running multiple models simultaneously.

However, when you naively average the weights of specialized models, the result is often disappointing. Instead of inheriting the strengths of every expert, the merged model frequently suffers from severe performance degradation, a phenomenon known as parameter interference or weight-space conflict.

To build better multi-task models, we must understand why weights clash and how modern techniques resolve these conflicts.

The Root Cause: Why Weights Clash

When a base model undergoes task-specific fine-tuning, its parameters drift away from their initial state along an optimization trajectory. The difference between the fine-tuned weights and the base weights is called a task vector.

If we visualize this process geometrically, parameter interference arises because different tasks push the model's parameters in divergent, often incompatible directions. As Yi-Lin Sung et al. observe in An Empirical Study of Multimodal Model Merging, models that have deeply diverged through downstream optimization do not naturally align when combined.

When we perform simple linear arithmetic on these divergent task vectors, two primary destructive phenomena occur:

  1. Sign Conflicts and Nullification: In Generalizability of Mixture of Domain-Specific Adapters from the Lens of Signed Weight Directions and its Application to Effective Model Pruning, Thai Le demonstrates that even for semantically similar tasks, optimization can yield parameters with opposing signs. When these weights are averaged, they cancel each other out. This weight nullification effectively erases the specialized knowledge that each model worked hard to acquire.
  2. Dominant Outliers and Noise Accumulation: Fine-tuning introduces both crucial structural changes and a large amount of optimization noise. MohammadReza Davari et al. show in Model Breadcrumbs: Scaling Multi-task Model Merging with Sparse Masks that high-magnitude weight updates act as dominant outliers that overwrite subtle but essential features of other tasks. Simultaneously, the additive accumulation of minor background updates introduces noise that eventually drowns out the clean signal of the learned features.

Ultimately, uniform weight averaging assumes that every parameter is equally important across all tasks. In reality, a weight that is critical for a coding model might be completely redundant or harmful for a translation model.

Strategies for Interference-Free Merging

To prevent specialized capabilities from washing out, recent research has shifted away from naive averaging toward interference-aware composition. These solutions broadly fall into three intuitive paradigms.

1. Sparsification and Masking

If parameter interference is caused by too many conflicting updates modifying the same parameters, the most direct solution is to make the updates sparse.

Le Yu et al. introduce this intuition in Language Models are Super Mario: Absorbing Abilities from Homologous Models as a Free Lunch. They demonstrate that you can drop between 90% and 99% of a fine-tuned model's delta parameters without sacrificing its performance. By aggressively sparsifying the task vectors before merging, the probability that two experts will attempt to modify the same parameter in conflicting directions drops drastically.

Similarly, the "Model Breadcrumbs" framework uses a band-pass sparse mask to filter out both the extreme, dominant outliers and the negligible optimization noise, leaving behind only the clean, essential "breadcrumbs" of task knowledge.

2. Disentanglement and Factorization

Another approach is to isolate the shared foundational knowledge from the task-specific updates so they do not corrupt one another.

In Twin-Merging: Dynamic Integration of Modular Expertise in Model Merging, Zhenyi Lu et al. argue that standard merging fails because it treats foundational weights and task-specific expertise as an inseparable unit. By explicitly disentangling these components, they prevent task-specific updates from overwriting the shared core capabilities of the base model.

A parallel concept is used by Wonyong Jeong et al. in Factorized-FL: Personalized Federated Learning with Parameter Factorization & Similarity Matching. To prevent "knowledge collapse" when averaging updates from incompatible client tasks, the authors factorize parameters into a globally shared knowledge base vector and a client-specific coefficient vector, ensuring that conflicting information is isolated to the task-specific coefficients.

3. Fine-Grained and Functional Alignment

Instead of treating the entire weight matrix as a monolith, we can look inside the layers to find functional groups of parameters that serve specific purposes.

For example, Wangyun Gu et al. propose NeuronMerge: Merging Models via Functional Neuron Groups. This method identifies groups of neurons within MLP layers that activate for specific domains (like math or coding). By identifying "general" neurons and selectively applying task arithmetic only where updates align, it discards conflicting task-specific updates that would otherwise cause spurious forgetting.

At the channel level, Mingyang Zhang introduces Channel Merging: Preserving Specialization for Merged Experts. By clustering and merging parameters at a finer, channel-grained level and pairing them with a task-specific router, the model preserves the unique expertise of individual adapters while maintaining a highly compact parameter footprint.

Naive Merging:
[Expert A Weights]  \
                     -->  [Simple Average]  -->  [Sign Cancellation & Noise]
[Expert B Weights]  /

Interference-Aware Merging:
[Expert A] -> [Sparsify / Disentangle / Group] \
                                                --> [Clean Combination] -> [Multi-Task Expert]
[Expert B] -> [Sparsify / Disentangle / Group] /

Moving Beyond Uniform Averaging

The evolution of model merging highlights a crucial realization in deep learning: the weight space of fine-tuned foundation models is remarkably redundant, yet highly sensitive to directional conflicts. By moving away from uniform linear averaging and adopting techniques like sparsification, parameter factorization, and functional neuron alignment, we can successfully bypass parameter interference—unlocking true multi-task composition without the cost of retraining.

History & citationVersion 2 · 2026-10-04

Current · Permanent link

  1. Version 2 (latest) · 2026-10-04 · 550 papers
  2. Version 1 · 2026-10-04 · 768 papers
Cite this direction

Latest version (a living page):

@misc{lacuna_parameter_interference_86919955,
  title = {parameter interference},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/parameter-interference/rdr_869199558d8b4d94a6f99e1f64a2ba95},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-19},
}

This exact version (version 2, frozen text):

@misc{lacuna_parameter_interference_86919955_v2,
  title = {parameter interference},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/parameter-interference/rdr_869199558d8b4d94a6f99e1f64a2ba95/v/2},
  note = {Version 2, frozen 2026-10-04. Accessed 2026-11-19},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake