Many teams now fine-tune a shared base model separately for different tasks and then try to combine the results into one model. The cheapest way to combine them is to add each task's weight change into the base weights. This "zero-overhead" merge leaves the network the same size and speed it started at. The catch is that the added changes interfere with each other, and adding them in ordinary Euclidean space distorts the angles between neurons. Orthogonal Model Merging shows the alternative works better: if each task's update is a rotation, the rotations can be averaged through their generators. A generator is a skew-symmetric matrix whose exponential is the rotation. The difficulty is cost. Rotations are expensive to maintain during training, and extracting them after the fact from ordinary fine-tunes took about 19 minutes per task with an exact matrix decomposition, and about 1 minute with an approximate iteration.
This note proposes to train each task as a rotation whose generator has low rank, and to store the generator in factored form. Each task's adapter is then already the object you want to average. Training is about as cheap as LoRA, a low-rank additive adapter. Merging is a sum of factored generators followed by a closed-form exponential that costs $O(d r^2)$ rather than $O(d^3)$. The result folds back into one weight matrix with no inference cost. The same factored form makes it cheap to compute a precise interference measure: how much two task rotations fail to commute. We propose this commutator as the test that separates the overlap between tasks that helps from the overlap that hurts, a question the literature currently disagrees about.
Core Claim
Write each adapted layer as $W = \exp(K)\,W_0$. Here $W_0$ is the frozen $d \times k$ weight and $K = UV^\top - VU^\top$ is a learned skew-symmetric generator, with $U, V \in \mathbb{R}^{d\times r}$. Let $Z = [U\ V]$ and $J = \begin{pmatrix}0 & I\\ -I & 0\end{pmatrix}$. Then $K = Z J Z^\top$, and the exponential has an exact small-core form:
$$\exp(K) = I + Z\,\varphi(J Z^\top Z)\,J Z^\top, \qquad \varphi(x) = \frac{e^x - 1}{x},$$
where $\varphi$ is evaluated on a $2r \times 2r$ matrix. The identity follows from expanding the power series of $ZJZ^\top$ and regrouping the factors.
The forward pass applies $W_0$ and then a rank-$2r$ correction, so the per-token cost matches LoRA. Gradients flow through ordinary matrix products, so plain AdamW works. That avoids the optimizer restriction An Embarrassingly Simple Way to Optimize Orthogonal Matrices at Scale reports for landing-style manifold steps, which fail with vanilla Adam.
Merging $T$ tasks forms $K_{\text{merge}} = \sum_t c_t K_t$, with magnitude-corrected weights $c_t$ in the style of OrthoMerge. This generator is again low rank, $2\sum_t r_t$, so the same formula gives the merged rotation. The merged rotation then collapses into $W_0$ at a cost of $O(d k \sum_t r_t)$. No SVD, matrix logarithm, or Procrustes step appears anywhere in the pipeline.
The claim has three parts. First, this adapter matches LoRA and orthogonal fine-tuning in single-task quality. Second, it merges as well as or better than OrthoMerge and null-space methods, with far less merge-time compute and no advance knowledge of the other tasks. Third, the normalized commutator $\|[K_i,K_j]\|_F / (\|K_i\|_F\|K_j\|_F)$ predicts per-task merge damage better than subspace overlap does.
Why the commutator is the right interference test
The research file shows the field split on whether overlap between tasks is harmful. Unraveling LoRA Interference: Orthogonal Subspaces for Robust Model Merging (OSRM) treats any cross-talk as damage. It forces each adapter's input projection away from the other tasks' features, which lifts the RoBERTa-large GLUE average from 70.04% to 76.59%. Rethinking Inter-LoRA Orthogonality in Adapter Merging runs the opposite test. Forcing outputs to be exactly orthogonal at runtime left the CLIP score unchanged at 0.9154 and lost the human preference test 81.3% of the time to direct merging. Efficient Orthogonal Fine-Tuning with Principal Subspace Adaptation (PSOFT) and MINGLE both relax strict orthogonality to recover plasticity.
Rotations give a cleaner criterion than either side uses. Two rotations whose generators commute compose exactly: the sum of generators equals the product of rotations, whatever the order. Disjoint subspaces, OSRM's goal, are one way to commute. Rotating in the same planes is another, and that is genuine sharing. The Baker–Campbell–Hausdorff expansion, the series that rewrites a product of exponentials as one exponential, places the first error of summing generators in the commutator term $\tfrac12[K_i,K_j]$. Our prediction is that harmful interference is the non-commuting part, not the overlap.
The commutator is cheap to compute. $\|[K_i,K_j]\|_F^2$ reduces to traces of products of the small Gram matrices $Z_i^\top Z_i$, $Z_i^\top Z_j$ and $Z_j^\top Z_j$, so the cost is $O(d r^2)$ per pair and needs no data.
Research Program
The first stage builds the adapter. We will implement the factored exponential with a numerically stable $\varphi$, using a Padé approximant or scaling-and-squaring on the $2r\times 2r$ core. We will also add an optional per-output-neuron magnitude vector, which merges by simple averaging, so that capacity can be tested separately from orthogonality. We will measure training throughput and memory against LoRA, PSOFT and BOFT. Both orthogonal-overhead papers in the file give the motivation here: PSOFT reports BOFT and qGOFT running out of memory on LLaMA-3B, and running 3.2x slower than PSOFT. Our adapter should sit within a small constant of LoRA's cost.
The second stage builds the merge operators. All three act on stored generators. The plain Lie sum uses OrthoMerge-style magnitude correction. The ordered product composes rotations, and the factored form keeps it exact by concatenating factors. The second-order BCH correction adds pairwise commutators. That correction is itself low rank, though at higher rank, and it tests whether the commutator is the damage term or merely a correlate. We will also implement continual merging, where tasks arrive one at a time and the running generator is extended. The merge cost per new task depends on $d$ and the total rank, not on $d^3$.
The third stage is analysis. For each task pair and layer we will record the normalized commutator, the subspace overlap between the column spaces of $Z_i$ and $Z_j$, and the cosine between weight deltas. We will then regress each task's accuracy drop after merging on these three quantities.
Baselines
The naive reference points are the unmerged base model and the individually fine-tuned task models. The first sets the floor and the second sets the ceiling. The simplest merge baseline is Task Arithmetic and TIES on standard LoRA adapters, folded into $W_0$. This is the zero-overhead default every practitioner would try first. OSRM reports 70.04% on the RoBERTa-large GLUE eight-task average for Task Arithmetic, and we will rerun it in our setup.
The strongest geometric competitor is Orthogonal Model Merging, applied two ways. The first applies it to models fine-tuned with orthogonal fine-tuning. The second applies it to LoRA fine-tunes through Orthogonal-Residual Decoupling, which uses the Newton–Schulz Procrustes variant. It is the strongest competitor because it already merges on the rotation group. Our question is whether native low-rank generators match it at lower cost. The file gives no accuracy numbers for it, so no reported result found on Lacuna for OrthoMerge accuracy. The file does give its extraction cost: roughly 19 minutes per task with SVD and about 1 minute with Newton–Schulz.
OSRM is the strongest interference-aware additive method, reported at 76.59% on RoBERTa-large GLUE. It is privileged because it sees every task's features before training. For continual merging, the competitor is Null-Space Filtering for Data-Free Continual Model Merging (NUFILT). It reports 83.6% on the eight-task ViT-B/32 sequence, which is 8.1 points above OPCM, and backward transfer of -1.5% versus -3.9% for WUDI-Merging on eight GLUE tasks with Flan-T5-base.
A further baseline is PSOFT adapters merged by Lie-averaging their $r\times r$ rotations inside the shared principal subspace of $W_0$. We expect this to be the real threat, discussed below.
All methods will use the same base checkpoints, task data and adapter parameter budget, matched by trainable parameter count. Each gets the same learning-rate sweep size per method; this matters because PSOFT reports sensitivity to learning rate above 8B parameters. Each gets the same merge-coefficient tuning protocol, either one held-out validation split or none at all. We will report merge wall-clock time on identical hardware.
Beating a baseline means one of two things. Either we exceed it in average merged accuracy at equal or lower merge cost, or we match it within noise across three seeds while removing a requirement it has: OSRM's look-ahead, NUFILT's per-layer SVD and optimization, or OrthoMerge's extraction step. Two ablations isolate the key ingredient. In the first we keep our trained adapters but merge them by Euclidean addition of $\exp(K_t)W_0 - W_0$. This separates the training parametrization from the merge geometry. In the second we freeze $U$ and $V$ to the principal singular directions of $W_0$, which tests whether task-specific rotation planes matter.
Why not just…?
Why not just add LoRA updates with Task Arithmetic or TIES? It is free and zero-overhead, and the evidence says it loses accuracy. OSRM's 70.04% versus 76.59% gap on GLUE and the 4–30% forgetting MINGLE reports for additive continual merging both show the damage. Our Euclidean-merge ablation shows whether the loss comes from the merge geometry or from the adapter type. If Euclidean merging of our rotation adapters already closes the gap, the Lie-algebra merge is unnecessary, and we would report that.
Why not just fine-tune normally and let OrthoMerge extract rotations afterward? This is the most direct competitor. Its cost grows with the number of tasks times a per-task decomposition. Its rotation is also a post-hoc fit to updates that were never trained to be rotations, so part of each update lands in a residual that is merged in Euclidean space. Our method stores the generator from the start. We will compare accuracy at equal parameter budget and the total merge time for 8 and 20 tasks. If OrthoMerge with Newton–Schulz matches our accuracy, our contribution narrows to cost and the continual setting, and we would say so.
Why not just train PSOFT adapters and Lie-average them in the shared principal subspace? This is a genuine threat. All PSOFT tasks rotate inside the same rank-$r$ subspace of $W_0$, so merging is an inexpensive $r\times r$ average. It is plausible this works well. Against it: a single shared subspace forces every task to compete for the same planes, which should raise the commutators between tasks. PSOFT's $\alpha,\beta$ relaxation vectors also break exact orthogonality, so the merge is no longer clean on the group. Three things settle it: the frozen-subspace ablation, a direct PSOFT-merge baseline, and the per-pair commutator statistics. If PSOFT-merge wins, the conclusion is that task-specific planes are not needed. That is a useful negative result, and it would redirect the project toward the commutator diagnostic alone.
Why not just use OSRM's look-ahead orthogonal initialization? It works when all tasks are known in advance. In a sequential or decentralized stream that assumption fails. We will report OSRM as an oracle upper bound and count matching it without look-ahead as success.
Proposed Experiments
We use three settings drawn from the file. The first is the eight GLUE tasks with RoBERTa-large in the OSRM setup, plus Flan-T5-base for the continual variant, using NUFILT's backward-transfer metric. The second is the eight-task CLIP ViT-B/32 sequential image-classification benchmark (SUN397, SVHN, MNIST, DTD, EuroSAT and others), for comparison against NUFILT's 83.6%. The third is LLM merging of math, code and instruction adapters on a Llama 3 family model, trained on NuminaMath-TIR and Magicoder-OSS-Instruct as OrthoMerge does and evaluated on MATH500, HumanEval+ and AGIEval. Metrics are average merged accuracy, normalized accuracy relative to the individual models, backward transfer in continual runs, training throughput and peak memory, and merge wall-clock time.
Three checks decide the project. First, single-task quality: if the low-rank rotation adapter falls more than about one point below LoRA on GSM8K and GLUE at matched parameters, even with the magnitude vector, the capacity cost kills the idea. Second, merge quality without look-ahead: the Lie-sum merge must beat Task Arithmetic and match OrthoMerge within noise, with merge time per task measured in seconds rather than minutes. Third, the commutator hypothesis, which is the one that would change how people think about interference. Across all task pairs and layers, the normalized commutator must predict post-merge accuracy drop with clearly higher rank correlation than subspace overlap or delta cosine.
The sharpest test of the third check uses deliberately constructed pairs. Some pairs overlap heavily but nearly commute, for example two tasks trained on related data from the same initialization. Others overlap little but do not commute. If the first kind merges well and the second merges badly, we have a data-free account of when interference helps. That account would reconcile OSRM with the MC-dropout result. If the BCH-corrected merge also recovers most of the damage on the second kind, the commutator is the cause of the damage and not just a correlate.
References
- Orthogonal Model Merging
- An Embarrassingly Simple Way to Optimize Orthogonal Matrices at Scale
- Unraveling LoRA Interference: Orthogonal Subspaces for Robust Model Merging
- Rethinking Inter-LoRA Orthogonality in Adapter Merging: Insights from Orthogonal Monte Carlo Dropout
- Efficient Orthogonal Fine-Tuning with Principal Subspace Adaptation
- MINGLE: Mixture of Null-Space Gated Low-Rank Experts for Test-Time Continual Model Merging
- Null-Space Filtering for Data-Free Continual Model Merging: Preserving Stability, Promoting Plasticity