TL;DR
Joint Null-Space Projection (JNSP) is a novel, data-free method for merging multiple fine-tuned models into a single multi-task model by treating interference as a geometric subspace problem. By projecting each task's weight updates onto the "joint null space" of its peers, JNSP achieves state-of-the-art performance on vision tasks—outperforming existing entry-wise methods like TIES-Merging by over 11 points—without requiring any training data, gradients, or optimization.
What Problem Does This Paper Address?
Foundation models are often fine-tuned into specialized "experts" for different tasks. However, deploying $N$ separate models is computationally expensive. Model merging aims to synthesize these experts into a single parameter vector.
The simplest approach, "Task Arithmetic," just adds the weight deltas (task vectors) together. Unfortunately, as the number of tasks increases, these updates begin to interfere with each other, leading to a significant drop in accuracy. Existing solutions like TIES-Merging or DARE attempt to fix this by filtering parameters entry-by-entry (e.g., by magnitude or sign voting).
The authors of this paper argue that these entry-wise methods miss the underlying geometry: weight updates are typically low-rank and act on specific low-dimensional subspaces of the hidden features. Interference isn't just about individual parameter values; it's about different tasks trying to "claim" the same input directions for different purposes.
Methods
The core idea of Joint Null-Space Projection (JNSP) is to identify the input directions each task cares about most and remove components that conflict with other tasks.
- Decompose the Delta: For every weight tensor in a fine-tuned model, the task vector $\tau_i$ is computed. Using Singular Value Decomposition (SVD), the method identifies the top-$r$ right-singular subspace ($S_i$)—the input directions along which that task moves the representation most.
- Define the Joint Null Space: For each task $i$, a "joint null space" ($N_i$) is defined as the orthogonal complement of the union of the subspaces claimed by all other tasks ($j \neq i$).
- Project and Sum: Each task vector is projected onto its respective joint null space. This "cuts away" the parts of the update that would interfere with its neighbors. The projected updates are then summed with the original pretrained weights.
Crucially, JNSP is closed-form, order-independent, and data-free. It requires no forward passes, no labels, and no optimization loops.
Datasets
The authors evaluate JNSP primarily on a standard eight-task image classification benchmark using CLIP backbones (ViT-B/16, ViT-B/32, and ViT-L/14):
- MNIST (digits), EuroSAT (satellite), GTSRB (traffic signs), SVHN (house numbers), DTD (textures), RESISC45 (aerial scenes), Stanford Cars, and SUN397 (scenes).
They also test the method on the GLUE benchmark (8 language tasks) using RoBERTa-large to explore its limits.
Key Insights and Ideas
- Interference is Geometric: The paper demonstrates that two weight deltas can have identical entry-wise statistics but completely different interference structures because they act on different input subspaces. This explains why entry-wise methods hit a performance ceiling.
- The "Conflict Map": The union of singular subspaces acts as a map of contested territory. JNSP is a "leave-one-out" strategy where each task gracefully yields directions claimed by any of its peers.
- Symmetry Matters: Unlike sequential projection methods common in continual learning (which are order-dependent and can "gut" later tasks), JNSP treats all tasks symmetrically.
- Data-Free Geometry: Remarkably, the geometric structure of the fine-tuned weights themselves is sufficient to predict and resolve interference without needing a single calibration image.
Key Results
On the ViT-B/16 backbone, JNSP achieves an average accuracy of 89.45% across eight tasks. This represents a massive leap over previous training-free baselines:
- +11.88 points over TIES-Merging (77.57%).
- +14.22 points over Task Arithmetic (75.23%).
- +0.98 points over WUDI’s consensus merge (88.47%), which is a much more complex method.
The gains are consistent across different model scales and even hold up on independently trained checkpoint releases (TALL). Qualitative analysis shows that JNSP often correctly classifies images where entry-wise baselines fail by predicting classes from the "wrong" task.
Limitations and Future Work
- Language Model Failure: In a surprising "negative result," JNSP performs poorly on RoBERTa-large for GLUE tasks, significantly trailing simple Task Arithmetic. The authors suggest that in language models, the "removed" directions might contain essential shared features that every task needs, unlike in vision encoders where updates are more task-specific.
- Hyperparameter $r$: While the method is relatively robust, it still requires choosing the rank $r$. Future work could explore adaptive, per-tensor rank selection.
- Generative Models: The method has not yet been tested on generative tasks like language modeling (LLMs) or image generation (Diffusion), where the nature of "interference" might be different.
Summary updated