Introduction
Deep learning faces a significant "storage and computation" challenge as the number of specialized models grows. While we have excellent models for specific tasks like cat classification or depth estimation, deploying them simultaneously is resource-intensive. Model merging offers a solution by fusing these "single-talent" models into a single "multi-talent" model without requiring shared initializations or costly fine-tuning.
The Challenge of Permutation Invariance
The primary obstacle in model merging is permutation invariance. Internal units (neurons) in two models may perform identical functions but appear in a different order. To merge them effectively, they must be aligned. Traditional methods rely on either weight similarity or activation similarity. However, as shown in Figure 1, high similarity in one space does not guarantee similarity in the other. Units with nearly identical weights can produce vastly different activations, creating "merging barriers" that degrade the performance of the fused model.
Methodology: MuDSC
The authors introduce MuDSC (Merging under Dual-Space Constraints), a framework that treats model merging as a dual-objective optimization problem.
- Dual-Space Similarity: MuDSC linearly combines similarity matrices from both weight and activation spaces. This ensures the matching algorithm identifies permutations that satisfy both constraints simultaneously, finding the "sweet spot" for merging.
- Iterative Alignment: Because layer weights depend on permutations of adjacent layers, MuDSC uses an iterative algorithm to refine alignment across the entire network until convergence.
- Group Alignment: To handle modern architectures like Vision Transformers (ViTs) and models with Group Normalization, MuDSC incorporates a group alignment strategy to manage complex internal and external permutation dependencies.
Experimental Results
MuDSC was evaluated across diverse benchmarks, including CIFAR-10/100, ImageNet-1k, and the Taskonomy dataset (covering 12 heterogeneous tasks).
As detailed in Table 1, MuDSC consistently outperforms state-of-the-art training-free baselines such as Git Rebasin and Zipit. On CIFAR-100 with ResNet20, MuDSC improved joint accuracy by over 1.3% compared to the best baseline.
Beyond homogeneous tasks, MuDSC excelled in merging models for radically different visual capabilities in the Taskonomy experiments, achieving the highest average scaled performance (84.84%). The method proved robust across various architectures, including ResNets, ViTs, and Swin Transformers.
Conclusion
MuDSC represents a significant step toward practical, off-the-shelf model fusion. By reconciling similarities in both weight and activation spaces, it effectively positions the merged model in a shared loss basin where performance for all individual tasks is preserved. While it requires a small amount of unlabeled data to compute activation similarities, its training-free nature and architecture-agnostic design make it a versatile tool for creating multi-talent neural networks.
Summary updated