Merging Fine-Tuned Models by Shrinking Eigen-Tasks

When several copies of one pretrained network are fine-tuned for different tasks, running all of them at test time multiplies memory and latency by the number of tasks. Model merging avoids this. It combines the copies' weight changes, called "task vectors", into one network that costs the same to run as a single model. The best training-free merges take each layer's weight change separately and compress it with a singular value decomposition, an SVD, which splits a matrix into ranked directions. They keep only the top few percent of directions. Two such methods are CART and Task Singular Vectors. Both still leave a gap of roughly five accuracy points to running the separate models, and both choose how many directions to keep with a hand-tuned cutoff.

This note proposes treating the stack of per-task weight changes for a layer as a three-way array (output dimension × input dimension × task) and denoising it with a tensor nuclear norm. A tensor nuclear norm is a convex penalty from signal recovery that favors arrays with low "rank" after a transform is applied along one axis. Here that axis is the task axis. The key observation is that, for merging, the transform along the task axis is the entire design. A poor transform makes tensor denoising collapse to plain weight averaging. The identity transform reduces it to per-task truncation, essentially CART or TSV-C. A transform built from the tasks' own similarity matrix gives something new: the method compresses "eigen-tasks", the principal combinations of tasks, rather than individual tasks. A single global threshold then decides how much rank each combination gets, in every layer at once.

Core Claim

Let $\Delta_m^{(l)} \in \mathbb{R}^{d_{out}\times d_{in}}$ be task $m$'s weight change in layer $l$, for $M$ tasks. Stack these changes into a tensor $\mathcal{W}^{(l)}$. Let $\Phi$ be an $M\times M$ orthogonal matrix applied along the task axis, and let $\hat S_k = \sum_m \Phi_{km}\Delta_m$ denote the $k$-th transformed slice. The closed-form proximal step of the transform-induced tensor nuclear norm soft-thresholds the singular values of every slice at a common level $\tau$. Here soft-thresholding means subtracting $\tau$ from each singular value and clipping at zero, written $\mathrm{SVT}_\tau$. Inverting the transform and summing over tasks, as task arithmetic does, gives the merged update

$$\Delta_{merged}^{(l)} \;=\; \sum_{k=1}^{M} c_k\,\mathrm{SVT}_\tau\big(\hat S_k\big), \qquad c = \Phi\mathbf{1},$$

where $\mathbf{1}$ is the all-ones vector. Two consequences follow directly. First, if $\Phi$ contains the normalized mean $\mathbf{1}/\sqrt{M}$ as a row, then $c$ is nonzero only on that row. In that case every contrast slice is thresholded and then discarded, and the merge reduces to a shrunken weight average. Second, if $\Phi$ is the identity, the merge is per-task soft truncation followed by summation. That is a soft-threshold version of CART and TSV-C.

Our proposal sets $\Phi$ to the eigenvectors of the $M\times M$ Gram matrix of task vectors, the matrix of their pairwise inner products, computed per layer or per layer type. This choice is permutation-equivariant: relabeling the tasks permutes the eigenvectors, so the merged model does not change. The usual Fourier transform used in tensor recovery lacks this property, because its circular structure assumes the third axis is ordered in time. The claim to test is this: with the same data-free compute and a matched parameter budget, compressing eigen-tasks under one global $\tau$ beats per-task truncation and stacked-matrix SVD, and the advantage grows with the number of tasks and how strongly they correlate.

Relation to Prior Work

The method comes from transform-induced tensor nuclear norms. Novel Methods for Multilinear Data Completion and De-noising Based on Tensor-SVD introduced the tensor SVD and its Fourier-based nuclear norm for video. Tensor Robust Principal Component Analysis with A New Tensor Nuclear Norm showed this norm is the convex envelope of tensor average rank and that its thresholding step splits into independent matrix SVDs across transformed slices. Low-Rank Tensor Completion With a New Tensor Nuclear Norm Induced by Invertible Linear Transforms extended the norm to any invertible transform. Tensor Completion by Multi-Rank via Unitary Transformation showed that data-driven unitary transforms, with a different rank per slice, need fewer samples. We reuse these tools unchanged. What differs is the object: these papers recover physical signals. They also flag orientation dependence as an open problem, and that problem becomes acute here because task order carries no meaning. The Gram-eigenvector transform is our answer to it for this setting.

The motivating evidence comes from matrix-level merging. Revisiting Weight Averaging for Model Merging (CART) centers task vectors on their average and truncates each layer's SVD. It finds a bell-shaped accuracy curve that peaks near an 8% rank ratio. Task Singular Vectors: Reducing Task Interference in Model Merging (TSV) shows that a 0.1 rank fraction keeps 99% of per-task accuracy. Its merge, TSV-M, decorrelates the tasks' singular vectors with an orthogonal Procrustes step. TSV also reports that interference is heaviest in early layers and leaves per-layer rank selection as future work. Both methods choose rank per task by a sweep. Our method instead shares rank across task combinations through one threshold. Per-layer ranks then emerge without any additional sweep, which is the open problem TSV names. Orthogonal Model Merging merges orthogonal components on a manifold, layer by layer. EMR-Merging: Tuning-Free High-Performance Model Merging keeps per-task 1-bit masks and scalars, so it is not a single static model.

One paper constrains the proposal rather than motivating it. Low-rank bias, weight decay, and model merging in neural networks argues that weight decay pushes tasks into nearly non-overlapping low-rank subspaces. If that holds, the Gram matrix is close to diagonal, the eigen-task transform is close to the identity, and our method reduces to per-task truncation. We treat this as a falsifiable prediction, not an obstacle. Gains should appear only in layers and task sets where the off-diagonal Gram mass is large. We will measure that mass directly.

Research Program

We will build the estimator above for every linear layer of CLIP vision transformers (ViT-B/32, ViT-L/14) and of RoBERTa-base. Compute is one SVD per transformed slice, the same order as CART and TSV-C, and needs no data or iterative solver.

We will study four transforms along the task axis: identity, Fourier under random task orderings, a mean-plus-contrast basis, and Gram eigenvectors. This tests the collapse prediction and the ordering-sensitivity prediction analytically and empirically. We will also try one global $\tau$ set by a data-free rule, keeping a fixed fraction of total transformed spectral energy across the whole model. We will report sensitivity to that one scalar against CART's rank sweep.

Two secondary extensions follow, and they will be dropped if they do not help. The first stacks same-shaped matrices across depth into a fourth axis, transformed by depth-wise Gram eigenvectors. The motivation is that output projections all write into the same residual stream, the shared running representation each block adds to, so they share coordinates. The second adds a sparse term solved by ADMM, as in robust tensor PCA, with $\lambda = 1/\sqrt{\max(n_1,n_2)n_3}$ from Tensor Robust Principal Component Analysis with A New Tensor Nuclear Norm. We expect this theory to be only a starting point here, since its incoherence and random-support assumptions are unlikely to hold for weight changes.

Baselines

The naive reference is task arithmetic from Editing models with task arithmetic, which adds scaled task vectors. TSV reports 70.79% for it on 8-task ViT-B/32. Plain weight averaging reaches 65.9% in CART, against a separate-model upper bound of 90.5%. We will rerun both from the same checkpoints, tuning only the global scaling coefficient on the same grid every method gets.

The strongest single-model, data-free competitor is TSV-M, at 85.86% mean accuracy on 8-task ViT-B/32 and 96.98% normalized accuracy on ViT-L/14 (TSV paper). CART, at 84.7% on ViT-B/32, is the second. We will rerun both from their released settings on identical checkpoints. Our method will get the same budget of hyperparameter evaluations that CART uses for its rank sweep, and every comparison will hold total retained rank fixed. To count as beating TSV-M, our method must exceed it by more than the seed-to-seed spread over three fine-tuning seeds on ViT-B/32. It must also show no loss on ViT-L/14.

AdaMerging+CART reaches 85.8% but uses unlabeled test data. We report it as a reference above the data-free regime. EMR-Merging reaches 93.7% on ViT-L/14 with per-task masks and is reported in a separate column, because it needs task identity at inference.

Three ablations isolate the transform. The identity transform under the same soft threshold separates the effect of the transform from that of soft versus hard truncation. A single SVD of all task changes stacked side by side tests whether a plain shared subspace is enough. Gram-eigenvector projection without slice thresholding tests whether the gain comes from compression across tasks alone.

Why not just…?

Why not just tune CART's rank ratio per layer? That is the most direct route to layer-adaptive rank. The two-layer-type sweeps in CART and TSV already show it costs validation passes that grow with depth. We will run a per-layer sweep as an oracle using validation data. If our single data-free threshold comes within one point of that oracle, the method earns its keep. If the oracle is far ahead, our threshold rule is the weak link, and we will say so.

Why not just concatenate the task matrices and take one SVD? This is a real threat. It captures a shared subspace across tasks with ordinary linear algebra, and TSV-M's whitening is closely related. Our formula shows the difference: the stacked SVD shares one set of directions, whereas eigen-task slices give different task combinations their own spectra. That difference matters only when the Gram matrix has strong off-diagonal structure. If the stacked SVD matches us at matched rank, the tensor view adds nothing, and we will report that outcome.

Why not just use EMR-Merging, which is already close to the separate models? EMR needs to know which task each query belongs to. Its storage also grows with the number of tasks. For mixed-task or unlabeled streams, a single static model is the right object, so EMR serves as a reference ceiling, not a competitor in the same regime.

Why not use the standard Fourier tensor nuclear norm? It depends on task order. Our ordering experiment will show how much accuracy shifts across random task orders.

Proposed Experiments

Benchmarks are the 8-task vision suite (Cars, DTD, EuroSAT, GTSRB, MNIST, RESISC45, SUN397, SVHN) on ViT-B/32 and ViT-L/14, the 14- and 20-task vision suites, and GLUE with RoBERTa-base. Infrastructure comes from the EMR_Merging codebase and the FusionBench baselines it wraps. For the 14- and 20-task suites, there is no reported result on Lacuna for TSV-M or CART, so we rerun them. Metrics are absolute and normalized accuracy, retained parameters, and merge wall-clock time.

Four checks are decisive. First, the mean-plus-contrast transform should reproduce weight averaging, and Fourier transforms should vary across task orderings while Gram eigenvectors stay fixed. Second, at matched rank, Gram eigenvectors must beat the identity transform and the stacked SVD. The prediction is that the margin grows from 8 to 20 tasks and concentrates in early layers, where TSV finds the most interference. Third, the per-layer gain must correlate with the measured off-diagonal Gram mass. This directly tests the near-orthogonality account from the low-rank-bias paper. Fourth, deliberately correlated subsets, such as the digit tasks MNIST and SVHN, should show the largest gains.

We will abandon the eigen-task approach if, on 8-task ViT-B/32, it fails to beat both the identity-transform ablation and the stacked SVD by more than the seed spread.