Saliency-Aware Model Merging

Jungin Park, Jiyoung Lee, Kwanghoon SohnICML 2026 · OpenReview

Abstract As the ecosystem of specialized foundation models expands, consolidating multiple task-specific models into a single, unified architecture has become a critical challenge. Current "data-free" model merging methods often rely on simple parameter-level heuristics that ignore inter-layer dependencies, leading to suboptimal performance. Saliency-Aware Merging (SA-Merging) introduces a novel framework that leverages structural connectivity scores and iterative pruning to identify essential parameter updates. By prioritizing end-to-end network pathways over isolated weight magnitudes, SA-Merging significantly outperforms existing methods across both vision and language tasks without requiring any training data.

The Challenge: Knowledge Silos and Localist Bias

In the era of foundation models, the standard paradigm involves fine-tuning a massive pre-trained model (such as LLaMA or CLIP) into various "experts" tailored for specific domains like medical text, code generation, or sentiment analysis. However, maintaining dozens of separate models is computationally expensive and creates isolated knowledge silos.

Model merging attempts to combine these experts back into a single model. The primary obstacle is task interference: when averaging the weights of different experts, they often conflict over the same parameters, degrading the merged model's performance.

Existing data-free merging methods—which operate without access to the original training data—typically rely on simple heuristics, such as retaining only the largest weight changes. This approach suffers from a localist bias. It treats every weight as an independent variable, ignoring the hierarchical nature of neural networks. For instance, a massive weight update in an early layer is functionally useless if the pathway to the output is bottlenecked by near-zero weights in subsequent layers.

The SA-Merging Framework

To overcome localist bias, SA-Merging shifts the focus from individual weight magnitudes to structural connectivity. The framework is built on four core pillars:

  1. Connectivity-Based Saliency: Drawing inspiration from structural pruning techniques like SynFlow, SA-Merging calculates a connectivity score ($R_n$). This score estimates the total signal transmission capacity of a task-specific update from the input all the way to the output, ensuring that only functionally impactful weights are preserved.
  2. Merge-Aware Modulation: To mitigate interference, the method evaluates whether experts agree on the direction of a weight change. If an expert's update contradicts the collective consensus of the ensemble, its saliency score is penalized. This acts as an implicit noise filter.
  3. Iterative Pruning: Rather than executing a one-shot merge, SA-Merging employs an iterative process (typically across 10 steps). In each step, it identifies and discards the least salient updates, progressively refining the model's functional backbone while preserving end-to-end connectivity.
  4. Rank-Wise LoRA Merging: Recognizing the prevalence of Low-Rank Adaptation (LoRA) in modern architectures, the authors extended SA-Merging to handle low-rank matrices. Instead of pruning individual matrix entries—which would destroy the low-rank structure—the method evaluates and prunes entire rank-1 components based on their contribution to network connectivity.

Key Insights

  • The "Blocked Path" Intuition: A parameter's value is only as useful as its context within the network. By utilizing the gradient of the connectivity score, SA-Merging successfully identifies weights situated on high-capacity pathways.
  • Data-Free Rivals Data-Assisted: Surprisingly, this purely structural, data-free approach frequently outperforms "test-time adaptation" methods that utilize unlabeled data during the merging process. This indicates that the internal structure of fine-tuned weights inherently contains enough information to resolve conflicts.

Empirical Performance

SA-Merging was rigorously evaluated across diverse benchmarks, demonstrating state-of-the-art results:

  • Vision Tasks: On an 8-task suite using CLIP ViT-B/32 backbones, SA-Merging achieved an 85.9% average accuracy. This surpasses the previous best data-free method, WUDI-Merging (85.2%), and vastly outperforms standard Task Arithmetic (69.1%).
  • Language Tasks: On the GLUE benchmark using RoBERTa-Base, the method reached 87.1% accuracy, representing a massive ~20% improvement over TIES-Merging (64.7%). It also showed strong performance on generative tasks using Flan-T5-base and Qwen-14B experts.
  • Efficiency: Despite its iterative nature, SA-Merging is highly efficient. Merging ViT-B/32 models took merely 20 seconds—roughly 6x faster than WUDI-Merging (though it does require about 3x more GPU memory to store connectivity gradients).
  • Conflict Reduction: Qualitative analysis confirmed that SA-Merging results in significantly fewer parameter conflicts compared to traditional methods.

Limitations and Future Directions

While highly effective, SA-Merging does introduce a memory overhead. Because the method tracks gradients and saliency scores for every expert during the iterative pruning process, GPU memory requirements scale linearly with the number of experts. This could pose a bottleneck when merging large ensembles of massive models on constrained hardware.

Additionally, the connectivity score serves as a structural surrogate for performance. While it proves highly effective empirically, it is not mathematically guaranteed to correlate perfectly with downstream accuracy across all conceivable architectures.

Future work may explore hybrid approaches that incorporate small amounts of calibration data, potentially closing the remaining performance gap between data-free merging and full multi-task fine-tuning.

Summary updated

Abstract

Model merging aims to consolidate multiple task-specific models fine-tuned on different datasets into a unified architecture that performs cross-domain proficiency. Current data-free model merging methods often struggle to scale as they rely on simple parameter-level heuristics that ignore inter-layer dependencies and non-uniform distribution of expertise. This work proposes SA-Merging, which is built upon connectivity-based saliency formulations from structural pruning (e.g., SynFlow) and extends them to the data-free model merging setting. We define a saliency score over task vectors relative to a shared base model, and further introduce merge-aware modulation that incorporates agreement across experts to mitigate task interference. Based on this formulation, an iterative saliency-aware merging procedure progressively removes non-informative updates while preserving end-to-end connectivity. Furthermore, we extend SA-Merging to introduce rank-wise saliency decomposition for LoRAs without compromising their structural integrity. Extensive experiments on vision and language tasks demonstrate the effectiveness of our saliency-based approach, further reducing the gap between data-free and test-time adaptation methods.
BibTeX
@inproceedings{park2026saliency,
  author = {Jungin Park and Jiyoung Lee and Kwanghoon Sohn},
  title = {{Saliency-Aware Model Merging}},
  booktitle = {International Conference on Machine Learning},
  year = {2026},
  url = {https://openreview.net/forum?id=IMKni29CXX},
}
Figure