Training-Free Pretrained Model Merging

Zhengqi Xu, Ke Yuan, Huiqiong Wang, Yong Wang, Mingli Song, Jie Song

Introduction

Deep learning faces a significant "storage and computation" challenge as the number of specialized models grows. While we have excellent models for specific tasks like cat classification or depth estimation, deploying them simultaneously is resource-intensive. Model merging offers a solution by fusing these "single-talent" models into a single "multi-talent" model without requiring shared initializations or costly fine-tuning.

The Challenge of Permutation Invariance

The primary obstacle in model merging is permutation invariance. Internal units (neurons) in two models may perform identical functions but appear in a different order. To merge them effectively, they must be aligned. Traditional methods rely on either weight similarity or activation similarity. However, as shown in Figure 1, high similarity in one space does not guarantee similarity in the other. Units with nearly identical weights can produce vastly different activations, creating "merging barriers" that degrade the performance of the fused model.

Figure 1: The inconsistency between weight and activation similarity. The "Best Merged" unit (star) exhibits high similarity in both spaces simultaneously.

Methodology: MuDSC

The authors introduce MuDSC (Merging under Dual-Space Constraints), a framework that treats model merging as a dual-objective optimization problem.

  1. Dual-Space Similarity: MuDSC linearly combines similarity matrices from both weight and activation spaces. This ensures the matching algorithm identifies permutations that satisfy both constraints simultaneously, finding the "sweet spot" for merging.
  2. Iterative Alignment: Because layer weights depend on permutations of adjacent layers, MuDSC uses an iterative algorithm to refine alignment across the entire network until convergence.
  3. Group Alignment: To handle modern architectures like Vision Transformers (ViTs) and models with Group Normalization, MuDSC incorporates a group alignment strategy to manage complex internal and external permutation dependencies.

Experimental Results

MuDSC was evaluated across diverse benchmarks, including CIFAR-10/100, ImageNet-1k, and the Taskonomy dataset (covering 12 heterogeneous tasks).

As detailed in Table 1, MuDSC consistently outperforms state-of-the-art training-free baselines such as Git Rebasin and Zipit. On CIFAR-100 with ResNet20, MuDSC improved joint accuracy by over 1.3% compared to the best baseline.

Table 1: Performance on CIFAR datasets. MuDSC consistently achieves the highest joint and per-task accuracy across different ResNet configurations.

Beyond homogeneous tasks, MuDSC excelled in merging models for radically different visual capabilities in the Taskonomy experiments, achieving the highest average scaled performance (84.84%). The method proved robust across various architectures, including ResNets, ViTs, and Swin Transformers.

Conclusion

MuDSC represents a significant step toward practical, off-the-shelf model fusion. By reconciling similarities in both weight and activation spaces, it effectively positions the merged model in a shared loss basin where performance for all individual tasks is preserved. While it requires a small amount of unlabeled data to compute activation similarities, its training-free nature and architecture-agnostic design make it a versatile tool for creating multi-talent neural networks.

Summary updated

Abstract

Recently, model merging techniques have surfaced as a solution to combine multiple single-talent models into a single multi-talent model. However, previous endeavors in this field have either necessitated additional training or fine-tuning processes, or require that the models possess the same pre-trained initialization. In this work, we identify a common drawback in prior works w.r.t. the inconsistency of unit similarity in the weight space and the activation space. To address this inconsistency, we propose an innovative model merging framework, coined as merging under dual-space constraints (MuDSC). Specifically, instead of solely maximizing the objective of a single space, we advocate for the exploration of permutation matrices situated in a region with a unified high similarity in the dual space, achieved through the linear combination of activation and weight similarity matrices. In order to enhance usability, we have also incorporated adaptations for group structure, including Multi-Head Attention and Group Normalization. Comprehensive experimental comparisons demonstrate that MuDSC can significantly boost the performance of merged models with various task combinations and architectures. Furthermore, the visualization of the merged model within the multi-task loss landscape reveals that MuDSC enables the merged model to reside in the overlapping segment, featuring a unified lower loss for each task. Our code is publicly available at https://github.com/zju-vipa/training_free_model_merging.
BibTeX
@inproceedings{xu2024training,
  author = {Zhengqi Xu and Ke Yuan and Huiqiong Wang and Yong Wang and Mingli Song and Jie Song},
  title = {{Training-Free Pretrained Model Merging}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2024},
}
Figure