Model merging has emerged as a powerful paradigm for creating multi-task systems without the prohibitive cost of joint retraining. By combining the weights of specialized "expert" models, researchers can produce a single "generalist" that inherits the capabilities of its parents. However, the field has long been bottlenecked by a strict requirement: the constituent models must share an identical neural architecture and, usually, a common pre-trained initialization.
Recent research is breaking this "identical architecture" constraint, developing methods to fuse knowledge across models with different widths, depths, and even fundamentally different structural designs.
The Permutation and Alignment Problem
The primary obstacle to merging any two neural networks is that internal representations are not naturally aligned. Even if two models are trained on the same data, their internal channels or neurons are often randomly permuted. Simple weight averaging fails because it attempts to combine neurons that represent entirely different features.
To solve this, researchers like Martin Jaggi and Sidak Pal Singh proposed Model Fusion via Optimal Transport. This approach treats model fusion as an alignment problem, using Optimal Transport (OT) to find a "soft-matching" between neurons based on their activation patterns or weight similarities. By re-aligning the coordinate systems of the models before merging, they demonstrated that it is possible to combine models even when they have different widths or were trained on heterogeneous data.
This concept was further extended to the Transformer architecture in Transformer Fusion with Optimal Transport. Authors Moritz Imfeld and Jacopo Graldi showed that by aligning multi-head attention and residual streams, one can merge independently trained Transformers into a single entity that, after brief fine-tuning, can outperform its parents.
Bridging Structural Gaps with Feature Zipping
While OT-based methods handle width differences, merging models from entirely different families—such as a Vision Transformer (ViT) and a ResNet—requires addressing deeper structural incompatibilities.
Daniel Bolya and George Stoica introduced ZipIt! Merging Models from Different Tasks without Training to tackle this. ZipIt! works by identifying redundant features across models and "zipping" them together. Crucially, it allows for "partial zipping," where models can share some layers while maintaining separate "heads" for different tasks. This enables the integration of specialized features from diverse model families into a single unified system, achieving significant performance boosts over traditional ensembling.
Similarly, Deep Neural Network Fusion via Graph Matching formulates the fusion as a graph matching problem. By aligning channels based on second-order weight similarities, the GAMF algorithm provides a scalable way to fuse networks that might not share a rigid layer-to-layer correspondence.
Network Growing and Knowledge Distillation
Another facet of this research direction focuses on "network growing"—the process of initializing a large model using the weights of smaller, pre-trained ones. Hanna Mazzawi and colleagues introduced Deep Fusion, which uses a fusion operator to accelerate the training of large models. This method reduces compute costs by up to 24% by effectively "recycling" the knowledge of smaller architectures into larger ones, even when the internal ratios of attention and feed-forward components differ.
When direct weight merging is too complex due to extreme architectural divergence, researchers turn to sophisticated distillation frameworks. UNIFORM, proposed by Weiming Zhuang and Chen Chen, distills knowledge from a diverse "model zoo" into a single student. By using voting mechanisms to resolve feature representation conflicts, UNIFORM can scale to over 100 teacher models with varying structures. This "fuse-then-merge" strategy is also seen in FUSIONCHAT by Fanqi Wan, which combines Chat LLMs of different scales (e.g., merging 7B models to rival 8x7B MoE models).
Collaborative Search and Swarm Intelligence
A more recent shift moves away from static weight averaging toward dynamic, search-based adaptation. Model Swarms, developed by Shangbin Feng and Zifeng Wang, treats a pool of expert models as particles in a swarm. Instead of using backpropagation, the swarm collaboratively searches the weight space to find optimal configurations. This approach is inherently more flexible than traditional merging, as it can discover latent capabilities within a heterogeneous ecosystem of models without requiring them to share a common ancestor or architecture.
The Path Forward
The ultimate goal of this research is to move beyond the "one-size-fits-all" constraint of current AI development. As noted in the survey by Wei Ruan and Tianming Liu, From Task-Specific Models to Unified Systems, the ability to integrate models with different structural designs remains one of the most significant open challenges in the field.
Whether through geometric alignment like EBM-CoT, which uses Equiangular Tight Frames to align task directions, or through data-efficient distillation like ProDistill, the field is converging on a future where the "model zoo" is not a collection of isolated silos, but a fluid resource that can be mixed and matched to create specialized, high-performance systems on demand.