Dynamic Fisher-weighted Model Merging via Bayesian Optimization

Sanwoo Lee, Jiahao Liu, Qifan Wang, Jingang Wang, Xunliang Cai, Yunfang Wu

Abstract

The fine-tuning of pre-trained language models has resulted in the widespread availability of task-specific models. Model merging offers an efficient way to create multi-task models by combining these fine-tuned models at the parameter level, without the need for training data or joint training on multiple datasets. Existing merging approaches typically involve scaling the parameters model-wise or integrating parameter importance parameter-wise. Both approaches exhibit their own weaknesses, leading to a notable performance gap compared to multi-task fine-tuning. In this paper, we unify these seemingly distinct strategies into a more general merging framework, and introduce Dynamic Fisher-weighted Merging (DF-Merge). Specifically, candidate models are associated with a set of coefficients that linearly scale their fine-tuned parameters. Bayesian optimization is applied to dynamically adjust these coefficients, aiming to maximize overall performance on validation sets. Each iteration of this process integrates parameter importance based on the Fisher information conditioned by the coefficients. Experimental results show that DF-Merge outperforms strong baselines across models of different sizes and a variety of tasks. Our analysis shows that the effectiveness of DF-Merge arises from the unified view of merging and that near-optimal performance is achievable in a few iterations, even with minimal validation data.

Introduction

In the current landscape of Large Language Models (LLMs), the ecosystem is populated by numerous "expert" models—pre-trained models fine-tuned for specialized tasks such as sentiment analysis, summarization, or code generation. While these models excel in their specific domains, the goal of creating a versatile multi-task model remains a significant challenge.

The standard approach to achieving multi-task capabilities is Multi-Task Learning (MTL), which involves training a single model on a combined dataset of all target tasks. However, MTL is often impractical due to its high computational cost and the requirement for simultaneous access to all original training data, which may be restricted by privacy or proprietary concerns.

Model merging has emerged as a promising "training-free" alternative. By combining the weights of existing expert models directly in the parameter space, researchers can create multi-task models without additional training. However, existing methods generally fall into two categories with distinct limitations:

  1. Model-wise scaling (e.g., Task Arithmetic): This approach applies a uniform scaling factor to all parameters of a model. While simple, it is often too coarse to handle the complex interactions between different tasks.
  2. Parameter-wise importance (e.g., Fisher Merging): This method weights parameters based on their calculated importance to a task. However, it typically uses static importance values derived at the end of fine-tuning, failing to account for how models interfere with each other during the merging process.
Conceptual comparison of merging strategies: Fisher Merging (parameter-wise), Task Arithmetic (model-wise), and the proposed DF-Merge which unifies both.

Methodology: Dynamic Fisher-weighted Merging (DF-Merge)

The authors propose DF-Merge, a framework that unifies model-wise scaling and parameter-wise importance into a single, cohesive objective.

The Unified Framework

DF-Merge defines a general merging function where each model is assigned a scaling coefficient ($\lambda$). The framework incorporates a covariance matrix, specifically the Fisher Information (FI), to weight individual parameters. This unification allows DF-Merge to leverage the flexibility of scaling while maintaining the precision of parameter-level importance.

Dynamic Fisher Information

A key innovation of DF-Merge is the use of Dynamic Fisher Information. Unlike static methods, DF-Merge estimates parameter importance at the scaled model state during each optimization step. This dynamic estimation helps identify "low-loss basins"—regions in the high-dimensional parameter space where the merged model maintains high performance across multiple tasks simultaneously.

Bayesian Optimization (BO)

Optimizing the scaling coefficients is non-trivial because performance metrics like accuracy are non-differentiable, precluding the use of standard gradient-based methods. DF-Merge employs Bayesian Optimization to search the coefficient space efficiently. By treating validation accuracy as a "black-box" function, BO uses a Gaussian Process to model the performance landscape and an acquisition function (such as Expected Improvement) to determine the most promising coefficients for the next iteration.

The DF-Merge workflow: (a) scaling models, (b) estimating dynamic parameter importance, and (c) merging, all within a Bayesian Optimization loop to maximize validation accuracy.

Efficiency and Data Requirements

To ensure computational feasibility, DF-Merge utilizes a diagonal approximation of the Fisher matrix. Furthermore, the process is highly data-efficient; parameter importance can be estimated using as few as 30 unlabeled validation samples per iteration.

Experimental Evaluation

The researchers evaluated DF-Merge using T5-base and T5-large architectures across six diverse NLP tasks, including paraphrase identification (PAWS), various question answering formats (QASC, QuaRTz, WikiQA), sentence completion (Story Cloze), and coreference resolution (Winogrande). DF-Merge was compared against several established baselines, including simple weight averaging, Fisher Merging, Task Arithmetic, TIES-Merging, and DARE.

Key Results and Insights

  • Performance Gains: DF-Merge achieved an average accuracy of 78.14% on T5-base, a significant 4.48 percentage point improvement over the best baseline, Task Arithmetic (73.61%).
  • Closing the MTL Gap: On the larger T5-large model, DF-Merge reached 83.59% average accuracy. This result substantially narrows the performance gap with the "Oracle" multi-task fine-tuned model (88.31%), demonstrating that merging can be a viable alternative to joint training.
  • Optimization Efficiency: Near-optimal performance is typically achieved within approximately 20 iterations of Bayesian Optimization. This efficiency suggests that the method can scale effectively as more models are added to the merge.
  • Robustness to Data Scarcity: Experiments revealed that using only 5% of the validation data is sufficient to outperform traditional merging methods, making DF-Merge practical for scenarios with limited labeled data.

Conclusion and Future Directions

DF-Merge represents a significant step forward in model merging by providing a unified view of scaling and importance weighting. While highly effective, the method currently requires models to share the same architecture and pre-trained initialization. Future research could explore merging heterogeneous architectures or developing entirely unsupervised variants that use the models themselves as pseudo-labelers for the optimization process.

Summary updated

Abstract

The fine-tuning of pre-trained language models has resulted in the widespread availability of task-specific models. Model merging offers an efficient way to create multi-task models by combining these fine-tuned models at the parameter level, without the need for training data or joint training on multiple datasets. Existing merging approaches typically involve scaling the parameters model-wise or integrating parameter importance parameter-wise. Both approaches exhibit their own weaknesses, leading to a notable performance gap compared to multi-task fine-tuning. In this paper, we unify these seemingly distinct strategies into a more general merging framework, and introduce Dynamic Fisher-weighted Merging (DF-Merge). Specifically, candidate models are associated with a set of coefficients that linearly scale their fine-tuned parameters. Bayesian optimization is applied to dynamically adjust these coefficients, aiming to maximize overall performance on validation sets. Each iteration of this process integrates parameter importance based on the Fisher information conditioned by the coefficients. Experimental results show that DF-Merge outperforms strong baselines across models of different sizes and a variety of tasks. Our analysis shows that the effectiveness of DF-Merge arises from the unified view of merging and that near-optimal performance is achievable in a few iterations, even with minimal validation data.
BibTeX
@inproceedings{lee2025dynamic,
  author = {Sanwoo Lee and Jiahao Liu and Qifan Wang and Jingang Wang and Xunliang Cai and Yunfang Wu},
  title = {{Dynamic Fisher-weighted Model Merging via Bayesian Optimization}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2025},
  url = {https://openreview.net/forum?id=hVLlDUSlfZ},
}
Figure