When we fine-tune a large pre-trained model for a specific task, we aren't changing every part of the model equally. Instead, the model learns a "task vector"—a specific direction of change in the high-dimensional parameter space. The challenge arises when we try to merge multiple specialized models into one. Simple techniques like weight averaging often fail because the specialized knowledge of one model acts as "noise" to another, a phenomenon known as task interference.
Recent research has shifted from treating these weight updates as flat lists of numbers to treating them as structured matrices that can be decomposed into their fundamental geometric components using Singular Value Decomposition (SVD).
The Low-Rank Nature of Task Knowledge
The core intuition behind this direction is that task-specific information is remarkably sparse. Even in a model with billions of parameters, the actual "knowledge" added during fine-tuning usually resides in a very low-rank subspace.
Antonio Andrea Gargiulo and Donato Crisostomi demonstrate this in their work on Task Singular Vectors (TSV). By applying SVD to the weight update matrices of a fine-tuned layer, they found that a tiny fraction of the singular vectors captures the vast majority of the task's performance. This allows for 10x compression of the task-specific update with almost no loss in accuracy.
Because these updates are low-rank, we can think of each task as a "thin" slice of the total parameter space. Interference happens when these slices overlap or point in conflicting directions.
Filtering Noise in Spectral Space
If task knowledge is concentrated in the top singular values, then the smaller singular values likely represent "noise" or artifacts of the fine-tuning process that don't contribute to the core task.
Yu-Ang Lee and colleagues introduced STAR (Spectral Truncation and Rescale) to exploit this. By moving weight updates into "spectral space," they filter out these low-impact components before merging. This "spectral cleaning" prevents the low-energy noise of ten different models from accumulating and drowning out the high-energy signals of the actual tasks. Similarly, Alireza Abdollahpoorrostam and the team at EPFL showed in MonoSoup that even a single fine-tuned checkpoint can be improved by adaptively reweighting these spectral components to favor task-specific directions over general-purpose ones.
Aligning Disparate Subspaces
A major hurdle in modern model merging is that many models are trained using Low-Rank Adaptation (LoRA). Since each LoRA model learns its own unique low-rank basis, they are often mathematically "incompatible" for direct averaging.
To solve this, George Stoica and Judy Hoffman proposed KnOTS (Knowledge Orientation Through SVD). KnOTS uses SVD to align the disparate weight subspaces of different LoRA models into a shared coordinate system. Once aligned, the models can be merged without the "personality loss" that usually occurs when forcing different mathematical representations together.
Aniello Panariello and others extended this concept with Core Space, demonstrating that we can project multiple task-specific updates into a unified, information-dense subspace. This allows for merging at a fraction of the computational cost of full-parameter methods while maintaining the union of all task-specific directions.
From Static Merging to Adaptive Subspaces
The most advanced application of this idea moves away from creating a single "average" model and toward a dynamic system that selects the right subspace for the right input.
In MASS (MoErging through Adaptive Subspace Selection), Donato Crisostomi and Alessandro Zirilli use SVD to isolate task-specific subspaces and then employ a training-free router to activate the relevant subspace at inference time. This effectively turns a collection of merged models into a Mixture-of-Experts (MoE) system without the need for expensive retraining.
This spectral perspective—treating model updates as a collection of singular vectors—transforms model merging from a messy process of "averaging weights" into a precise operation of "composing subspaces." By isolating the most significant directions of change, researchers like Baijiong Lin and James Kwok in BYOM have shown we can build multi-task models "for free," preserving the unique personality of every specialist within a single, efficient generalist.