Abstract: Averaging the parameters of models that have the same architecture and initialization can provide a means of combining their respective capabilities. In this paper, we take the perspective that this "merging" operation can be seen as choosing parameters that approximately maximize the joint likelihood of the posteriors of the models' parameters. Computing a simple average of the models' parameters therefore corresponds to making an isotropic Gaussian approximation to their posteriors. We develop an alternative merging procedure based on the Laplace approximation where we approximate each model's posterior as a Gaussian distribution whose precision matrix corresponds to its Fisher information. We first show that our "Fisher merging" technique provides a performance boost in settings where simple parameter averaging is currently used -- specifically, robust fine-tuning and model ensembling. Then, we compare merging to standard gradient-based transfer learning and demonstrate that merging enables a fundamentally different method for transferring capabilities across models. Specifically, we show that Fisher merging is competitive with gradient-based transfer learning approaches (while being significantly cheaper) in intermediate-task training and domain-adaptive pre-training. We also show that our merging procedure makes it possible to combine models in previously unexplored ways. We release our code to facilitate future research into methods for merging models.
Introduction
In modern machine learning, the ability to combine the specialized capabilities of multiple models is highly desirable—for instance, merging a general language model with one specialized in a specific domain like medicine. The standard approach, transfer learning, involves fine-tuning a pre-trained model on a new task using gradient descent. However, this process is computationally intensive and often suffers from "catastrophic forgetting," where the model loses its proficiency on original tasks while learning new ones.
Model merging offers a cheaper alternative by directly averaging the weights of models that share the same architecture and initialization. Until now, this has typically been done via "isotropic merging," a simple unweighted average. However, isotropic merging assumes all parameters are equally important for every task, a simplification that limits performance.
Methods: Fisher-Weighted Averaging
This paper frames model merging as a probabilistic inference problem. The goal is to find parameters that maximize the joint likelihood of the posteriors of the models being merged.
- Isotropic Merging (Baseline): A simple average that assumes parameter importance follows a uniform, circular (isotropic) Gaussian distribution.
- Fisher Merging: This method uses the Laplace approximation to represent each model's posterior as a Gaussian distribution. The importance, or "precision," of each parameter is determined by the diagonal of the Fisher Information Matrix.
In the final merged model, each parameter is a weighted average of the corresponding parameters in the original models. If a weight has high Fisher information in Model A but low in Model B, the merged model prioritizes Model A’s version. This is a closed-form solution; once Fisher information is calculated (requiring only a single pass over data), merging is instantaneous and requires no further training.
Experimental Evaluation
The authors evaluated Fisher merging across diverse benchmarks, including NLP (GLUE) using BERT and RoBERTa, and Computer Vision (ImageNet) using ViT-B/16 to test robust fine-tuning on out-of-distribution (OOD) sets.
Key Results
- Efficiency: Fisher merging is significantly more efficient than traditional methods, proving roughly 6x cheaper than standard fine-tuning for intermediate tasks and up to 275x cheaper than isotropic merging when considering the total pipeline for multiple merges.
- Performance and Robustness: In NLP tasks like RTE and MRPC, Fisher merging consistently outperformed isotropic merging. In vision tasks, it provided a superior trade-off between standard ImageNet accuracy and OOD robustness compared to existing state-of-the-art averaging methods.
- Ensembling: Merging five BERT models achieved performance parity with traditional prediction ensembling but resulted in a single model that was 5x faster during inference.
Discussion and Limitations
Fisher merging provides a rigorous mathematical foundation for weight averaging and introduces the "donor" model concept, where knowledge from an intermediate task can be merged into a target model without sequential fine-tuning. This effectively sidesteps catastrophic forgetting.
However, the method currently requires that all models share the exact same pre-trained checkpoint (initialization). Additionally, to maintain computational feasibility, the authors utilize a diagonal approximation of the Fisher matrix. Future work may explore more complex approximations (like Kronecker-factored matrices) or methods to merge models with different task-specific heads.
Summary updated