Teams increasingly fine-tune one shared foundation model many times, once per task, using small add-on weight updates called LoRA adapters (low-rank adaptation: the base weights stay frozen and each task learns a thin pair of matrices whose product is added to each layer). Users then want a single model that does all the tasks. The ideal way to combine adapters is to distill them, training one adapter to reproduce every task adapter's predictions on that task's inputs. Distillation needs the task data, though, and that data is often private or gone. So practitioners average the weights instead, and the averages interfere with each other.
This note argues that the right data-free merge is distillation done in closed form. Near each task model, the distillation loss (the divergence between the task model's predictions and the merged model's) is well approximated by a quadratic whose matrix is the Fisher information, a measure of how much the outputs move when a weight moves. Kronecker-factored curvature (K-FAC), which stores that Fisher as a product of an input-covariance matrix and an output-gradient-covariance matrix per layer, makes the quadratic cheap to store and solve. The proposal is to ship each adapter with its two small curvature factors per layer as a standard "adapter card." Merging, rank compression, and deciding which layers are safe to merge then become one optimization: second-order distillation that never touches data.
Core Claim
Let $\Delta_t = B_t A_t$ be task $t$'s LoRA update at a layer. Let $S^{in}_t$ be the covariance of that layer's inputs on task $t$, and $S^{out}_t$ the covariance of the gradients flowing back to its outputs, with labels sampled from the task model (the true Fisher). To second order, the expected KL divergence from task model $t$ to a model using merged update $\Delta$ at that layer is $\tfrac12\,\mathrm{tr}\big(S^{out}_t(\Delta-\Delta_t)S^{in}_t(\Delta-\Delta_t)^\top\big)$. This follows because the Fisher is the Hessian of the KL at zero discrepancy, and K-FAC factorizes it as $S^{in}_t \otimes S^{out}_t$. Summing over tasks gives the dataless distillation objective
$$\min_{\mathrm{rank}(\Delta)\le r}\ \sum_{t=1}^{N} \mathrm{tr}\big(S^{out}_t(\Delta-\Delta_t)S^{in}_t(\Delta-\Delta_t)^\top\big).$$
Without the rank constraint, its minimizer solves the generalized Sylvester equation $\sum_t S^{out}_t\,\Delta\,S^{in}_t=\sum_t S^{out}_t\,\Delta_t\,S^{in}_t$, which conjugate gradient handles using only matrix products. With the constraint, alternating least squares over the two LoRA factors solves exact linear subproblems.
The residual value of the objective is itself useful. It is a predicted distillation loss for merging that layer, and so a principled, data-free test of whether two blocks are compatible. The file lists the lack of exactly such a test as an open problem in modular merging.
The claim is falsifiable. If the output-gradient factor $S^{out}_t$ adds nothing over input covariance alone, or if the quadratic does not track the real KL divergence after merging, the idea reduces to existing activation-covariance methods.
Relation to Prior Work
Two papers supply the method. Dataless Weight Disentanglement in Task Arithmetic via K-FAC (TAK) shows that pre-computed K-FAC factors let a system be data-aware without data at merge time. It uses them as a regularizer during full-weight fine-tuning, reporting 88.3% against naive task arithmetic's 80.2% on ViT-B/16. TAK never treats LoRA products, never imposes a rank constraint on the result, and does not interpret the quadratic as a distillation loss. Learning in the Fisher Subspace (FILet) shows that K-FAC Fisher factors predict which LoRA directions are safe to change for a single adapter. We reuse its factor estimation and its trick of computing in a reduced basis, applied to many adapters at once.
Three papers motivate the gap. Merging Models with Fisher-Weighted Averaging is the same Laplace-style idea, but with a diagonal Fisher on full models and no rank control. Unraveling LoRA Interference (OSRM) argues that interference comes from adapters acting on other tasks' inputs. It uses input covariance only, and it must see other tasks' features before training. Our objective contains that input term but merges after the fact and adds output sensitivity. Decouple and Orthogonalize (DO-Merging) is purely weight-based, and the OSRM and FILet evidence suggests weight-only geometry misses data sensitivity.
Two further papers constrain the design. A Geometric View of Model Merging shows that averaging LoRA factors fails because of scaling symmetries between $B$ and $A$. Our objective operates on the product $\Delta$, so it is symmetry-invariant by construction. Its high-rank lifting ran out of memory on Llama-3 8B, which warns us to keep curvature factors compressed. Navigating the Accuracy-Size Trade-Off (FlexMerge) builds merged models of fractional size by greedily merging blocks according to weight cosine similarity. We replace that heuristic with the predicted KL residual.
On the distillation side, KD-LoRA trains student adapters with a KL loss on real data. That is the data-using upper reference our closed form approximates. m2mKD shows that distilling module by module beats monolithic distillation for modular networks, which supports doing distillation per layer.
Research Program
We first define the adapter card. It holds the LoRA factors plus, for each adapted layer, $S^{in}_t$ and $S^{out}_t$. For wide layers these are stored as a top-$k$ eigendecomposition plus a scaled identity, so a card costs roughly what a few extra LoRA ranks would. The factors are collected in one pass over the task's training data when the adapter is trained, as TAK does. TAK reports four minutes for the whole 8-vision suite.
The merge engine then reads only cards and outputs a merged adapter at a requested rank $r$. That rank may be higher than any input adapter's rank, which gives a lifted merge. We also build a curvature-guided variant of FlexMerge that greedily merges the block pair with the smallest predicted KL increase.
The analysis has two parts. First, we test whether the per-layer quadratic predicts true post-merge KL divergence. Second, we test where K-FAC's block-diagonal assumption breaks, since drift compounds across layers when each layer is merged independently. If compounding is severe, we test a cheap correction that rescales each layer's weight in the objective by its measured downstream sensitivity. That correction stays data-free because it uses only quantities stored on the card.
Baselines
The naive reference is task arithmetic: summing or averaging the LoRA products with a single tuned scaling coefficient. It is the minimum any method must beat.
The strongest published data-free competitor on LoRA vision merging is DO-Merging, which reports 77.88% average accuracy on ViT-B/32 against 74.06% for task arithmetic, per Decouple and Orthogonalize. GeoMerge with lifting is next, at 77.10% normalized accuracy on the same backbone per A Geometric View of Model Merging. On 6NLI with Llama-3 8B, GeoMerge reports 91.88% against task arithmetic's 90.38%, which makes it the bar there. Both methods will be rerun from the same LoRA checkpoints, with the same held-out validation set for any hyperparameter tuning and equal search budgets.
OSRM is the strongest data-aware competitor. It reports moving RoBERTa-large GLUE task arithmetic from 70.04% to 76.59%. Because it alters training, we will train its adapters with the same data and steps as ours.
Diagonal Fisher merging is the cheap curvature baseline. It uses the same Fisher pass as our method, so the comparison is exact. For it, and for our method on 6NLI, no reported result for LoRA was found on Lacuna.
The data-using ceiling is multi-teacher KD-LoRA-style distillation: each task adapter acts as teacher on its own data, and a single student adapter of rank $r$ is trained with matched steps. KD-LoRA reports retaining about 97% of full fine-tuning performance in its single-task setting.
The key ablations are $S^{out}_t=I$ (input covariance only), $S^{in}_t=I$, and diagonal factors, all solved with the same rank-constrained solver.
Beating the field means exceeding DO-Merging and GeoMerge by a margin larger than seed variance on vision and 6NLI. It also means closing at least half the gap between the best data-free method and the data-using distillation ceiling.
Why not just…?
Why not just use input covariance alone? This is the most serious threat. An objective that weights errors only by where each task's inputs lie, with $S^{out}_t=I$, is simpler and may capture most of the benefit, as OSRM's success hints. Our claim needs the output-gradient factor to matter. That would show up when a task's sensitive output directions differ from other tasks' sensitive output directions, for example when classification heads read different subspaces. The $S^{out}_t=I$ ablation settles this directly. If the gain is negligible, we will report the method as a rank-constrained, symmetry-invariant activation-covariance merge and drop the second-order framing.
Why not just use a diagonal Fisher? It is linear in memory and needs no solver. However, diagonal approximations discard the within-layer correlations that the Kronecker Laplace work of Ritter et al. and Daxberger et al. found important. The interaction of a low-rank product with its inputs is exactly such a correlation. The direct comparison, run at matched Fisher-estimation cost, will show whether this matters for adapters.
Why not just distill with whatever data remains? When data is available, real distillation should win. We treat it as the ceiling rather than a rival. We also test our closed-form merge as an initializer for a short data-based distillation run, which would make the method useful in both regimes.
Why not just apply TAK's regularizer to LoRA training? That requires coordinating training across tasks, which defeats post-hoc modular composition. We will still run it as a reference on the vision suite to measure what the post-hoc constraint costs.
Proposed Experiments
We train rank-16 LoRA adapters on CLIP ViT-B/32 and ViT-B/16 for the eight vision tasks, on Llama-3 8B for 6NLI, and on RoBERTa-large for GLUE, and collect a card for each. Metrics are average and normalized accuracy, card storage, and merge wall-clock time. All baselines above are run from identical checkpoints.
The first decisive check measures, per layer and task, the rank correlation between predicted and actual KL divergence on held-out task data. A correlation below about 0.5 would undermine the premise.
The second decisive check is the factor ablation on vision and 6NLI.
The third compares curvature-guided against cosine-guided FlexMerge at 1×, 2×, and 4× size on the vision suite, using the FlexMerge code. FlexMerge reports a 13.5% accuracy jump from 1× to 2× with cosine ordering. We ask whether curvature ordering reaches the same accuracy at a smaller size.
The fourth sweeps the merged rank $r$ from 16 to 128, to locate where lifting stops paying off and how close the closed form comes to the data-using distillation ceiling.
References
- Dataless Weight Disentanglement in Task Arithmetic via Kronecker-Factored Approximate Curvature
- Learning in the Fisher Subspace: A Guided Initialization for LoRA Fine-Tuning
- Merging Models with Fisher-Weighted Averaging
- Unraveling LoRA Interference: Orthogonal Subspaces for Robust Model Merging
- Decouple and Orthogonalize: A Data-Free Framework for LoRA Merging
- A Geometric View of Model Merging: Quotient Fréchet Averages from Toy Models to LoRA
- Navigating the Accuracy-Size Trade-Off with Flexible Model Merging
- KD-LoRA: A Hybrid Approach to Efficient Fine-Tuning with LoRA and Knowledge Distillation
- m2mKD: Module-to-Module Knowledge Distillation for Modular Transformers
- Online Structured Laplace Approximations for Overcoming Catastrophic Forgetting
- Laplace Redux - Effortless Bayesian Deep Learning