Joint Null-Space Projection for Data-Free Model Merging

Tian XiaarXiv 2026 · arXiv

TL;DR

Joint Null-Space Projection (JNSP) is a novel, data-free method for merging multiple fine-tuned models into a single multi-task model by treating interference as a geometric subspace problem. By projecting each task's weight updates onto the "joint null space" of its peers, JNSP achieves state-of-the-art performance on vision tasks—outperforming existing entry-wise methods like TIES-Merging by over 11 points—without requiring any training data, gradients, or optimization.

What Problem Does This Paper Address?

Foundation models are often fine-tuned into specialized "experts" for different tasks. However, deploying $N$ separate models is computationally expensive. Model merging aims to synthesize these experts into a single parameter vector.

The simplest approach, "Task Arithmetic," just adds the weight deltas (task vectors) together. Unfortunately, as the number of tasks increases, these updates begin to interfere with each other, leading to a significant drop in accuracy. Existing solutions like TIES-Merging or DARE attempt to fix this by filtering parameters entry-by-entry (e.g., by magnitude or sign voting).

The authors of this paper argue that these entry-wise methods miss the underlying geometry: weight updates are typically low-rank and act on specific low-dimensional subspaces of the hidden features. Interference isn't just about individual parameter values; it's about different tasks trying to "claim" the same input directions for different purposes.

Methods

The core idea of Joint Null-Space Projection (JNSP) is to identify the input directions each task cares about most and remove components that conflict with other tasks.

  1. Decompose the Delta: For every weight tensor in a fine-tuned model, the task vector $\tau_i$ is computed. Using Singular Value Decomposition (SVD), the method identifies the top-$r$ right-singular subspace ($S_i$)—the input directions along which that task moves the representation most.
  2. Define the Joint Null Space: For each task $i$, a "joint null space" ($N_i$) is defined as the orthogonal complement of the union of the subspaces claimed by all other tasks ($j \neq i$).
  3. Project and Sum: Each task vector is projected onto its respective joint null space. This "cuts away" the parts of the update that would interfere with its neighbors. The projected updates are then summed with the original pretrained weights.
Figure 1. JNSP in three moves. (1) One pretrained CLIP encoder is fine-tuned once per task, so the experts share a coordinate system and need no alignment. (2) A task’s update inside a weight tensor concentrates on a few input directions (a cap); where caps overlap, tasks interfere, and entrywise rules cannot see it. JNSP projects each task off the union of the other tasks’ caps, keeping only directions no neighbour claims, symmetrically and in no fixed order. (3) The projected updates sum to one model serving every task, with no images, labels, gradients or optimisation.

Crucially, JNSP is closed-form, order-independent, and data-free. It requires no forward passes, no labels, and no optimization loops.

Datasets

The authors evaluate JNSP primarily on a standard eight-task image classification benchmark using CLIP backbones (ViT-B/16, ViT-B/32, and ViT-L/14):

  • MNIST (digits), EuroSAT (satellite), GTSRB (traffic signs), SVHN (house numbers), DTD (textures), RESISC45 (aerial scenes), Stanford Cars, and SUN397 (scenes).

They also test the method on the GLUE benchmark (8 language tasks) using RoBERTa-large to explore its limits.

Key Insights and Ideas

  • Interference is Geometric: The paper demonstrates that two weight deltas can have identical entry-wise statistics but completely different interference structures because they act on different input subspaces. This explains why entry-wise methods hit a performance ceiling.
  • The "Conflict Map": The union of singular subspaces acts as a map of contested territory. JNSP is a "leave-one-out" strategy where each task gracefully yields directions claimed by any of its peers.
  • Symmetry Matters: Unlike sequential projection methods common in continual learning (which are order-dependent and can "gut" later tasks), JNSP treats all tasks symmetrically.
  • Data-Free Geometry: Remarkably, the geometric structure of the fine-tuned weights themselves is sufficient to predict and resolve interference without needing a single calibration image.

Key Results

On the ViT-B/16 backbone, JNSP achieves an average accuracy of 89.45% across eight tasks. This represents a massive leap over previous training-free baselines:

  • +11.88 points over TIES-Merging (77.57%).
  • +14.22 points over Task Arithmetic (75.23%).
  • +0.98 points over WUDI’s consensus merge (88.47%), which is a much more complex method.

The gains are consistent across different model scales and even hold up on independently trained checkpoint releases (TALL). Qualitative analysis shows that JNSP often correctly classifies images where entry-wise baselines fail by predicting classes from the "wrong" task.

A qualitative comparison showing JNSP correctly identifying images where TIES-Merging and Task Arithmetic fail due to task interference.

Limitations and Future Work

  • Language Model Failure: In a surprising "negative result," JNSP performs poorly on RoBERTa-large for GLUE tasks, significantly trailing simple Task Arithmetic. The authors suggest that in language models, the "removed" directions might contain essential shared features that every task needs, unlike in vision encoders where updates are more task-specific.
  • Hyperparameter $r$: While the method is relatively robust, it still requires choosing the rank $r$. Future work could explore adaptive, per-tensor rank selection.
  • Generative Models: The method has not yet been tested on generative tasks like language modeling (LLMs) or image generation (Diffusion), where the nature of "interference" might be different.

Summary updated

Abstract

Merging independently fine-tuned checkpoints into a single multi-task model requires no data, no labels, and no gradient steps. In practice the task vectors of different tasks interfere, and the standard remedies--averaging, magnitude pruning, sign election--operate entry by entry, ignoring that each delta induces a low-dimensional update on the hidden features it acts upon. We recast merging as a subspace interference problem. For every two-dimensional weight tensor we take the top-r right-singular subspace of each task's delta--the input directions along which that task moves the representation most--and remove from each task vector the component lying in the union of the other tasks' subspaces. The resulting Joint Null-Space Projection (JNSP) is closed-form and order-independent, needs no forward pass, and is defined entirely by the checkpoint set. On eight CLIP image-classification tasks JNSP merges ViT-B/16 experts to 89.45% average accuracy without using a single image, 11.88 points above the strongest parameter-wise training-free baseline (TIES-Merging, 77.57%), 14.22 above Task Arithmetic, and 0.98 above WUDI's data-free consensus merge (88.47%). The gain holds across three backbones on the independently trained TALL release (+10.8, +12.6, +9.1 over TIES for ViT-B/16, ViT-B/32, ViT-L/14) and replicates there at 89.20%; controls confirm the source--a single shared subspace instead of per-task ones loses 2.9 points, and a sequential variant collapses to 69.6%. On RoBERTa-large the same projection underperforms plain task arithmetic, which we report rather than hide. Code is available at https://github.com.
BibTeX
@misc{xia2026joint,
  author = {Tian Xia},
  title = {{Joint Null-Space Projection for Data-Free Model Merging}},
  year = {2026},
  eprint = {2611.17103},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.17103},
}
Figure