Abstract
Neural network merging often fails due to weight permutations that place models in different basins of the loss landscape. Existing pairwise, layer-wise alignment methods suffer from cycle inconsistency and high variance. We present $C^2M^3$ (Cycle-Consistent Multi-Model Merging), a data-free approach that aligns multiple models into a shared "universe" space. By enforcing global cycle consistency and employing simultaneous layer optimization via the Frank-Wolfe algorithm, $C^2M^3$ eliminates drift and variance. Experimental results across various architectures and datasets demonstrate significant performance gains, including a 20% accuracy improvement in multi-model aggregation on CIFAR-100.
1. Introduction
Deep learning models trained from different initializations typically reside in distinct basins of the loss landscape. While functionally similar, their internal weight permutations differ, making simple weight averaging ineffective. Prior methods like Git Re-Basin attempt to align these models pairwise and layer-by-layer. However, this approach is prone to cycle inconsistency—where sequential alignments do not return to the origin—and high variance due to stochastic optimization orders.
2. Methodology
$C^2M^3$ addresses these issues by shifting from pairwise alignment to a "hub-and-spoke" architecture. Every model is mapped to a common, shared "universe" space $U$. This design ensures that any pairwise alignment, derived as a composition of maps to the universe, is inherently cycle-consistent.
To optimize these permutations, $C^2M^3$ utilizes the Frank-Wolfe algorithm to update all layers simultaneously, capturing inter-layer dependencies missed by local heuristics. The process is entirely data-free, relying on weight similarity. For deep architectures like ResNets, the authors incorporate REPAIR to reset activation statistics after merging, which is vital for maintaining performance.
3. Results
The proposed method was evaluated on MNIST, CIFAR-10/100, and ImageNet. A key finding is that models mapped to the universe space exhibit strong linear connectivity, with loss barriers effectively vanishing.
In multi-model merging scenarios (e.g., 5 models), $C^2M^3$ significantly outperformed baselines. On CIFAR-100 with ResNet20, it achieved 46% accuracy, doubling the performance of the previous state-of-the-art. Furthermore, $C^2M^3$ achieved zero drift in cyclic tests and zero variance across different optimization seeds, providing a deterministic and robust merging solution.
4. Conclusion and Future Work
$C^2M^3$ provides a robust framework for multi-model merging by ensuring cycle consistency and global optimization. While it shows great promise for federated learning and model aggregation, the quality of the merge can be limited by outlier models that are difficult to align. Future work will focus on scaling the global optimization to larger models and addressing sensitivity to divergent training regimes.
Summary updated