Model merging has emerged as a computationally efficient alternative to multi-task learning, allowing researchers to combine specialized "expert" models into a single multi-functional system without the prohibitive cost of retraining. However, the field currently faces a significant bottleneck: the architectural homogeneity constraint. Most existing techniques require that the models being fused share an identical neural architecture and, often, a common pre-trained initialization.
The Shared Ancestry Requirement
The success of weight-averaging techniques, such as Fisher Merging Matena et al. and RegMean Jin et al., relies on the concept of linear mode connectivity. This principle suggests that models starting from the same pre-trained checkpoint reside in the same low-loss basin; therefore, a linear interpolation of their weights will likely result in a model that retains the capabilities of both parents.
As noted by Rame et al., this dependency on a shared ancestor prevents the integration of models trained from scratch independently or those belonging to distinct architectural lineages. For instance, current optimization strategies like PSO-Merging Zhang et al. are generally restricted to combining models within the same family, such as two Llama-3 variants, and cannot bridge the gap between structurally different families like Llama and Mistral.
Why Heterogeneity Breaks Merging
The constraint is not merely a matter of parameter counts; it is rooted in how neural networks represent information. Even when architectures are identical, independently trained models suffer from permutation variance. During training, internal channels and neurons are ordered randomly. Fusing these models directly is akin to trying to merge two jigsaw puzzles where the pieces have the same shapes but represent different parts of the image.
Furthermore, Hammoud et al. highlight that this structural lock-in prevents the synthesis of expertise across models of different sizes or scaling configurations. If one model uses a specific recurrent unit and another uses a transformer-based attention mechanism, there is no direct mathematical mapping to average their weights Byrne et al..
Strategies for Cross-Architecture Integration
To move beyond these constraints, researchers are exploring three primary directions:
1. Structural Alignment and Graph Matching Instead of assuming neurons are already aligned, some methods attempt to find the optimal permutation before merging. Liu et al. propose treating model fusion as a graph matching problem, aligning channels based on second-order weight similarities. This allows for the fusion of networks that were trained independently, though it still typically requires the layer counts and widths to match.
2. Latent Space Aggregation A more radical approach is to ignore the weights entirely and focus on the representations the models produce. Relative Latent Space Aggregation (RLSA), proposed by Crisostomi et al., merges the latent spaces of different networks into a unified "atlas." Because this method operates on output representations rather than internal parameters, it is inherently architecture-agnostic, enabling collaboration between heterogeneous models.
3. Modular Composition and Recycling Rather than blending weights, some frameworks focus on "recycling" models as modular components. MixtureKit Chamma et al. provides a pipeline to compose heterogeneous models into a Mixture-of-Experts (MoE) architecture. In this setup, a router decides which specialized model to invoke for a given input, bypassing the need to mathematically average their internal structures. Similarly, Abbasnejad et al. use task vectors with learned scaling to adapt models, though they note that composing these vectors across different families remains a significant challenge.
The Path Forward
The inability to merge models with different internal dimensions or layer configurations remains a primary obstacle to creating truly universal AI systems. While techniques like DARE Yu et al. can reduce interference by sparsifying weights, they still operate within the bounds of a shared initialization. As the "model zoo" of specialized experts continues to grow, the development of methods that can map and combine parameters across fundamentally different structural designs—what the authors call "cross-architecture merging"—represents the next frontier in model integration.