Merging Fine-Tuned Models by Rotating a Shared Subspace

When several copies of one pre-trained model are fine-tuned on different tasks, practitioners often want to fold them back into a single set of weights. The goal is one model with exactly the original shape and speed, so that serving it costs nothing extra. The two families of methods that do this best both pay for it. Low-rank adapters (LoRA, small add-on matrices whose product is added to a frozen weight) are cheap to train, but their task updates collide when summed. The strongest fixes then run a full singular value decomposition (SVD) or a Procrustes alignment (finding the rotation that best maps one matrix onto another) on every large layer after training. Orthogonal fine-tuning is the other family. It adapts a model by rotating its weights instead of adding to them, which keeps the angles between neurons intact and has been argued to merge more gracefully. The price is slow training and a merge step that works on large rotation matrices.

This note proposes putting the orthogonal structure where it is cheapest: a small subspace that every task already shares. Each task rotates only within the top-$r$ singular directions of the frozen pre-trained weight. Those directions are fixed by the base model and are identical for all tasks, with no data exchange needed. As a result, every task's adaptation is a small $r \times r$ rotation acting in the same coordinates. Merging reduces to averaging a handful of tiny matrices in closed form, with no large SVD or Procrustes step. The merged model collapses into a single weight matrix of the original size. The same structure yields a free diagnostic that predicts, layer by layer, when combining rotations will interfere. That diagnostic lets us test directly the open question of whether orthogonality actually helps merging.

Core Claim

Two parts of the literature point to this idea without combining it. Efficient Orthogonal Fine-Tuning with Principal Subspace Adaptation (PSOFT) restricts the orthogonal rotation to the principal subspace of the frozen weight, parameterizes it with a Cayley transform, and adds small scaling vectors to relax strict norm preservation. It reports gains of +2.3% over LoRA on GSM8K and training up to 3.2x faster than block-diagonal OFT. However, PSOFT only studies single-task adaptation and says nothing about merging. Orthogonal Model Merging (OrthoMerge) merges orthogonal fine-tunes by averaging their rotations in the Lie algebra, the flat "tangent" space of the rotation group, with a correction that prevents averaged rotations from shrinking toward identity. For ordinary fine-tunes, it first extracts a rotation from each full-size layer by Procrustes, which takes 19 minutes per task with exact SVD or about 1 minute with Newton-Schulz iterations. It operates purely after training, on full-dimensional matrices.

Our observation is this. Every task is fine-tuned from the same base weight $W_0 = U \Sigma V^\top$, so the top-$r$ left singular vectors $U_r$ are a common, data-free coordinate system. We write each task's adapted layer as

$$W_i = W_0 + U_r\,(D_i R_i - I)\,U_r^\top W_0,$$

where $R_i \in O(r)$ is an $r \times r$ rotation learned for task $i$, $D_i$ is a diagonal scaling that plays the role of PSOFT's relaxation, and $I$ is the identity. The merged layer uses the same form with a single $\bar R$ and $\bar D$. We obtain $\bar R$ by magnitude-corrected averaging of $\log R_i$ as in OrthoMerge, followed by an $r \times r$ matrix exponential. We obtain $\bar D$ by averaging the scalings separately, which gives structurally the magnitude-direction split that Decouple and Orthogonalize (DO-Merging) recovers by post-hoc optimization. The whole merge costs $O(r^3)$ per layer rather than $O(d^3)$, and the output is a single dense matrix of the original shape.

The shared coordinates also make interference measurable. The Baker–Campbell–Hausdorff expansion says that composing two rotations $e^{X}e^{Y}$ differs from $e^{X+Y}$ by $\tfrac12[X,Y]$ plus higher-order terms. Here $[X,Y]=XY-YX$ is the commutator, which is zero exactly when the two rotations can be applied in either order with the same result. We therefore define the per-layer, per-pair interference score

$$c_{ij} = \frac{\|[\log R_i, \log R_j]\|_F}{\|\log R_i\|_F\,\|\log R_j\|_F},$$

which costs a few $r \times r$ multiplications. We claim that $c_{ij}$ predicts where averaged rotations lose task accuracy. If that holds, it gives a principled, data-free rule for which layers to merge by rotation averaging and which to merge additively or leave at the base weights.

Research Program

The first component is the adapter and its training. We implement the shared-subspace rotation adapter with two ways of keeping $R_i$ orthogonal. One is PSOFT's Cayley map, which needs an $r \times r$ inverse. The other is the retraction-free "leap and land" update of An Embarrassingly Simple Way to Optimize Orthogonal Matrices at Scale (POGO). POGO keeps matrices on the orthogonal manifold using five matrix products per step and scaled to hundreds of thousands of constraints.

We are candid about the role of POGO. At $r \le 256$, a Cayley inverse is cheap, so POGO is not the headline. It matters when $r$ grows, or when we use several subspace blocks per layer, and we treat it as an ablation. The training-overhead story that does matter is the comparison with Parameter Efficient Quasi-Orthogonal Fine-Tuning via Givens Rotation (qGOFT). qGOFT reports roughly 3.3–3.5x slower batches because it chains many sparse rotations. The rotation-overhead baseline for merging is OrthoMerge's post-hoc Procrustes. If approximate orthogonality is used anywhere, we follow Beyond the Ideal: Analyzing the Inexact Muon Update, which shows that less precise orthogonalization requires smaller learning rates and more momentum. We re-tune those two knobs jointly rather than reusing Adam defaults.

The second component is the closed-form merge and the commutator diagnostic described above. It includes a mixed rule: layers whose mean $c_{ij}$ exceeds a threshold fall back to additive merging of $U_r(D_iR_i - I)U_r^\top W_0$, or are left unmerged at $W_0$. We tune that threshold once on held-out tasks and then freeze it.

The third component treats the ongoing orthogonality debate as a hypothesis to test. Rethinking Inter-LoRA Orthogonality found that forcing adapter outputs to be orthogonal left CLIP score unchanged at 0.9154 in compositional image generation. Human raters preferred plain additive merging 81.3% of the time, which suggests overlap between adapters can be constructive. When Shared Knowledge Hurts (SVC) and Task-Driven Subspace Decomposition (LoDA) likewise argue that the failure is over-counting shared directions, or blocking useful shared ones, rather than a lack of orthogonality.

Our design deliberately forces all tasks into one shared subspace, which is the opposite of the separation strategies in Task Singular Vectors (TSV-M) and Unraveling LoRA Interference (OSRM). If shared-subspace rotations merge well on discriminative benchmarks, that supports the constructive-sharing view outside generation, and $c_{ij}$ tells us where it breaks. If they merge badly, the commutator analysis should show why.

Baselines

The naive reference point is Task Arithmetic: standard LoRA adapters, collapsed by summing $\lambda \sum_i B_iA_i$ into $W_0$ (LoRA), with $\lambda$ tuned on the same validation split we use. On CLIP ViT-B/32 with eight vision tasks, DO-Merging reports 74.06% for Task Arithmetic on LoRA adapters. TSV reports 70.79% for Task Arithmetic on fully fine-tuned models. OSRM reports 70.04% average on GLUE with RoBERTa-large.

The strongest data-free competitors on LoRA checkpoints are DO-Merging (77.88% on ViT-B/32 LoRA merging) and SVC. SVC is training-free spectral rescaling, reports +13.0% over Task Arithmetic on eight vision tasks and +6.1 points on GSM8K with Llama-2-7B, and runs in minutes. For full fine-tunes, the strongest is TSV-M at 85.86% on ViT-B/32 across eight tasks. For the data-dependent regime, it is OSRM at 76.59% on GLUE. OSRM gets access to other tasks' hidden states, which our method does not use, so we report it separately as an upper-reference rather than a like-for-like competitor.

The most direct competitor is OrthoMerge, applied to two kinds of checkpoints: ordinary LoRA fine-tunes via Procrustes decoupling, and our own rotation checkpoints via its Lie-algebra averaging. We found no reported numeric result on Lacuna for OrthoMerge on these benchmarks, so we will run it ourselves with both the SVD and the Newton-Schulz variants.

For fairness, every method uses the same base checkpoints and the same task data. Adapter trainable-parameter counts are matched within 10%. Each method gets the same per-task training budget in GPU-hours and the same number of trials over merge hyperparameters on a shared validation split. Beating the field means two things. On the PEFT merging benchmarks, the method must exceed DO-Merging and SVC in merged absolute accuracy, and not merely in normalized accuracy. It must also match or beat OrthoMerge's accuracy at a merge cost at least an order of magnitude lower.

Three ablations isolate the key ingredient. The first rotates within a random shared $r$-dimensional subspace instead of the principal one. The second gives each task its own principal subspace computed from its fine-tuned weights, which removes the shared coordinates. The third merges the rotation adapters by plain Euclidean averaging of $R_i$ rather than Lie-algebra averaging.

Why not just…?

Why not just train ordinary LoRA and apply SVC or TSV-M afterward? This is the most serious threat. These methods need no change to training, and SVC runs in minutes. Our answer cannot be speed alone. It must be that adapters trained to rotate in shared coordinates merge more accurately, and more stably as the number of tasks grows. STAR (STAR) reports that TIES collapses past five to seven merged models, and the 8→14→20 task sweep below tests whether our advantage appears at scale. If SVC on LoRA matches us at 20 tasks, the honest conclusion is that the structure buys only a cheaper merge, and we will report that.

Why not just run OrthoMerge with Newton-Schulz on ordinary fine-tunes? At about one minute per task, the post-hoc cost is modest. What it cannot do is make the fine-tune be a rotation: the Procrustes residual still gets merged in Euclidean space. The same paper also observes that approximate alignment often matches exact alignment, which suggests the rotation it extracts is loosely defined. Running OrthoMerge on the same tasks from the same base weights decides whether training in rotation form improves merged accuracy beyond what post-hoc extraction recovers.

Why not just use PSOFT and average its adapters directly? If averaging $R_i$ entrywise, or averaging Cayley parameters, works as well as magnitude-corrected Lie averaging, the merging contribution shrinks to a bookkeeping observation. The Euclidean-average ablation measures exactly this. A Euclidean average of rotations is not itself a rotation, so we also record its deviation from orthogonality.

Why not just freeze a random orthonormal down-projection per task, since random subspaces in high dimensions are nearly orthogonal anyway? Weight-space near-orthogonality is cheap. OSRM's analysis, however, shows that the harmful cross-talk is $B_jA_jh_i$, meaning one task's adapter responding to another task's inputs. Random subspaces do not suppress that term. We include frozen random-$A$ LoRA, in the style of SLAO's orthogonal $A$ initialization (Merge before Forget), as a baseline so that this is measured rather than asserted.

Proposed Experiments

The primary vision setting is CLIP ViT-B/32 and ViT-L/14 merged across the standard 8-, 14- and 20-task image classification suites used by TSV, DO-Merging and SVC. We report absolute accuracy, accuracy normalized to each individual fine-tune, and the individual fine-tunes' own accuracy, since a weaker single-task adapter could inflate normalized scores. The language settings are the eight GLUE tasks with RoBERTa-large, matching OSRM, and Llama-2-7B merges evaluated on GSM8K, matching SVC.

We add one continual setting, the eight-task ViT-B/32 continual merge, where NUFILT reports 83.6%. There we apply the closed-form merge incrementally as each task arrives, using a running Lie-algebra mean. Efficiency is measured as training throughput and peak memory against LoRA, PSOFT-Cayley and qGOFT; merge wall-clock against OrthoMerge (SVD and Newton-Schulz), TSV-M and SVC; and inference latency. Latency should be identical to the base model, and we verify this rather than assume it.

The decisive checks are three. First, on the PEFT benchmarks at 20 tasks, shared-subspace rotations must beat DO-Merging and SVC on LoRA in absolute merged accuracy while merging at least ten times faster than OrthoMerge. Second, the commutator score $c_{ij}$ must predict per-layer merge damage. We measure damage by the accuracy drop from swapping one layer's merged weights back to a single task's weights, and we require a rank correlation clearly above what a plain weight-cosine score achieves. The commutator-gated mixed merge must also outperform uniform rotation averaging. Third, the random-subspace and per-task-subspace ablations must each lose accuracy relative to the shared principal subspace. If they do not, the benefit comes from rotation form alone, not from shared coordinates, and the paper's claim narrows accordingly.