When we want to combine the capabilities of two specialized neural networks—say, one that writes code and another that summarizes medical papers—the traditional approach is joint multi-task training. However, retraining is computationally expensive and often impossible if the original datasets are private or unavailable.
A growing body of research suggests a more efficient alternative: instead of training a merged model through gradient descent, we can calculate its optimal weights using closed-form analytical solutions. By treating model merging as a mathematical optimization problem rather than a learning problem, researchers are achieving near-perfect knowledge fusion in milliseconds.
Moving Beyond Simple Weight Averaging
The simplest way to merge models is "Weight Averaging" (WA), where you simply take the mean of the parameters. While WA works for models that have stayed close to their common initialization, it often fails as models specialize. This is because neural network weights don't live in a simple linear space; averaging them often leads to "interference" that destroys the specialized capabilities of both parents.
To solve this, researchers like Xisen Jin and Xiang Ren proposed RegMean (Dataless Knowledge Fusion by Merging Weights of Language Models). They reframed merging as a linear regression problem. Instead of averaging weights, they seek a merged weight matrix that minimizes the L2 distance between the merged model’s predictions and the individual experts' predictions. Because this objective is a quadratic form, it has a closed-form solution—a single matrix calculation that replaces hours of training.
Aligning Representations Instead of Parameters
A key intuition driving this field is that we shouldn't just care about the values of the weights, but the representations they produce. If a merged model produces internal activations similar to the original experts, it will likely retain their performance.
This "representation alignment" is the core of several recent breakthroughs:
- Minimizing Feature Drift: Wenju Sun and colleagues introduced LOT Merging (Towards Minimizing Feature Drift in Model Merging), which uses the Moore-Penrose inverse to calculate weights that minimize the discrepancy in internal features. This allows for state-of-the-art fusion using as few as 16 data samples.
- Representation Correction: Jialin Wu and Jiajun Wen developed ReACT (From Parameter to Representation), which treats model interference as a linear distortion. By solving a convex quadratic function, they can "correct" the merged model's representations near-instantly.
- Submodule Linearity: While entire LLMs are non-linear, Rui Dai and Sile Hu found that internal submodules (like MLP blocks) behave remarkably linearly (Leveraging Submodule Linearity Enhances Task Arithmetic Performance in LLMs). They use this property to derive optimal, layer-specific merging weights analytically.
Mathematical Engines of Fusion
The shift toward analytical merging relies on a specific toolkit of matrix algebra. For example, in the domain of Graph Neural Networks (GNNs), Vipul Garg and Sayan Ranu developed GNNMerge (GNNMerge: Merging of GNN Models Without Accessing Training Data). They show that because message-passing operations are linear transformations post-aggregation, the merging objective can be solved using Frobenius norm minimization. This results in a 136x speedup over retraining.
Similarly, for Transformer models, Binchi Zhang and Zaiyi Zheng leverage Singular Value Decomposition (SVD) to align models across rotation symmetries (Beyond the Permutation Symmetry of Transformers). By finding the optimal rotation matrix analytically, they can align independently trained models into the same "low-loss basin" before merging.
Dataless and Multi-Modal Composition
Perhaps the most powerful aspect of these analytical approaches is their ability to work without the original training data. MetaGPT, developed by Yuyan Zhou and Bingning Wang, calculates merging coefficients based solely on the geometry of the model weights (MetaGPT: Merging Large Language Models Using Model Exclusive Task Arithmetic). They use the squared magnitude of task vectors as a proxy for task importance, allowing for "dataless" fusion.
This principle extends to complex generative tasks. In Custom Diffusion, Nupur Kumari and Jun-Yan Zhu use Lagrange multipliers to solve a closed-form optimization that allows multiple user-defined concepts (like a specific pet and a specific art style) to be composed into a single image-generation model without joint training (Multi-Concept Customization of Text-to-Image Diffusion).
By replacing iterative optimization with exact matrix solutions, these methods are transforming model merging from a trial-and-error heuristic into a precise, scalable engineering discipline.