Shared Balanced Initialization for Alignment-Free Federated LoRA

When many devices fine-tune a large model together without sharing data, the cheapest thing to send is a low-rank update. Each client learns two thin matrices, $B$ and $A$, whose product $BA$ is its change to a weight matrix. This is LoRA, or low-rank adaptation. The server then averages the clients' $B$ matrices and $A$ matrices separately. That averaging is broken in a well-documented way. The same product can be written in many factor forms, since $(BG)(G^{-1}A)=BA$ for any invertible $G$, so averaging factors can cancel updates that agree in substance.

Current fixes are explicit. One rotates each client's factors to a reference before averaging. Another adds a penalty that pulls the product toward a global target. A third freezes one factor in each round. We propose to remove the cause rather than correct the effect. Work on implicit regularization, meaning the constraints plain gradient descent imposes on its own path, shows two things. Gradient flow on a factorization conserves the matrix $B^\top B - AA^\top$. Small initialization makes the factors grow along the dominant shared directions of the target. Together these suggest that if every client starts from the same small, balanced factor pair, client factors stay on a common surface and in nearly a common orientation. Plain factor averaging then becomes nearly exact, at no client-side cost.

Core Claim

The claim has three parts. First, factor-averaging error in federated LoRA has two separable parts. One is an orthogonal rotation mismatch, which Procrustes alignment addresses. The other is a scale or shear imbalance between $B$ and $A$, which rotation alignment cannot touch. Second, the standard LoRA initialization ($B=0$, $A$ random) creates both parts. A shared small balanced initialization, combined with a cheap server-side rebalancing step at each aggregation, suppresses both. Third, the remaining rotation mismatch should scale with how much the clients' dominant gradient subspaces disagree, not with rank or step count. We will state and test this as a quantitative prediction.

What would be new is the use of a conserved quantity that all clients share as an aggregation-compatibility guarantee. Alignment would come from initialization and from server-side projection, not from client-side rotation or a tuned penalty.

Mechanism

Under gradient flow on any loss of the product $BA$, with $B\in\mathbb{R}^{d\times r}$ and $A\in\mathbb{R}^{r\times k}$, the following quantity is conserved:

$$\frac{d}{dt}\left(B^\top B - AA^\top\right)=0.$$

This is the matrix form of the layer-balancing invariant proved in Algorithmic Regularization in Learning Deep Homogeneous Models: Layers are Automatically Balanced. Suppose all clients start from the same pair $(B_0,A_0)$. Then every client, whatever its data, moves on the same surface $\{B^\top B - AA^\top = C_0\}$. With standard LoRA initialization, $C_0=-A_0A_0^\top$, which is maximally unbalanced. Clients reach different products through different amounts of hidden scale imbalance, and that imbalance is part of the gauge freedom that wrecks averaging. With a balanced start, $C_0\approx 0$, each client's factors take the form $B_i=U_iS_i^{1/2}Q_i$ and $A_i=Q_i^\top S_i^{1/2}V_i^\top$. The only freedom left is an orthogonal $Q_i$.

The small-initialization analyses of matrix completion explain why $Q_i$ should also be shared:

In that phase the factors evolve almost linearly. Each client's factors are roughly the shared initialization amplified along the top singular directions of its own gradient. If clients' top subspaces largely coincide, the amplified components, and hence the $Q_i$, coincide too. The mismatch then grows with subspace disagreement.

The quantity we track is the aggregation gap $\Delta=\bar B\bar A-\frac1M\sum_i B_iA_i$, where $M$ is the number of clients and bars denote factor means. When clients share a start, this equals $-\frac1M\sum_i \delta B_i\,\delta A_i$, with $\delta B_i=B_i-\bar B$ and $\delta A_i=A_i-\bar A$. The gap is second order in client deviations. It is large only when deviations are large along directions that leave the product unchanged, and those are exactly the directions the invariant and the shared growth phase control.

Real clients use AdamW and discrete steps, which break the invariant. This is a known open problem in the implicit-regularization literature. We therefore do not rely on the invariant holding exactly. At each round the server averages products exactly in a reduced space, using the QR-plus-small-SVD construction of FedPA-LoRA. It then re-factors the result as a balanced pair. Finally it picks the remaining orthogonal freedom by a single Procrustes fit to the averaged factors, computed on the server.

Clients restart from this pair. Their own factors were already nearly balanced and nearly aligned, so the restart should move them very little. That directly attacks the "initialization mismatch" FedPA-LoRA identified for SVD re-broadcast.

Research Program

The program has three stages.

Controlled theory in federated low-rank matrix sensing. We use the synthetic setting from the small-initialization literature: $100\times100$ matrices of true rank 3, search ranks from exact to heavily overspecified. Clients see disjoint, unequally sized entry masks, or Gaussian Orthogonal Ensemble measurements with client-specific perturbations of the target, which gives tunable heterogeneity. We aim to prove a bound on $\|\Delta\|_F$ after one round of $T$ local steps. The bound should be linear in the initialization scale $\alpha$ plus a term in the principal angle between clients' top-$r$ subspaces. The proof would reuse the early-phase growth analysis of Ma et al.. The leave-one-out "shadow trajectory" technique of Chi et al. is the natural tool, with "leave one client out" in place of "leave one sample out."

A secondary question is whether rank-overspecified clients still recover the true structure under federated averaging. Centralized theory says the rank of small-initialized gradient descent adapts on its own. This question matters because heterogeneous client ranks are currently handled by truncation heuristics.

The algorithm on real models. The full method is shared seed, small balanced initialization, unmodified local training, and server-side exact averaging with balanced re-factorization. Clients run vanilla LoRA training and send the usual $B$ and $A$, so per-round communication is identical to standard federated LoRA.

Diagnosis. We will decompose every method's aggregation gap, round by round, into a rotation part and a balance part. The rotation part is the gap removed by optimal per-client Procrustes rotation. The balance part is what remains after removing the imbalance $\|B_i^\top B_i - A_iA_i^\top - C_0\|$. This decomposition tests the core claim directly, separately from accuracy.

Baselines

Naive reference: FedIT. FedIT, or naive federated LoRA, averages factors from standard LoRA initialization. FedRot-LoRA reports it at 76.80% MNLI accuracy with RoBERTa-Large and 50 clients, and naive FedLoRA at 28.77% HumanEval pass@1 with Llama 3-8B. Any credible method must clear these by a wide margin.

Strongest direct competitor: FedRot-LoRA. It aligns each client's factors to the previous global factors by orthogonal Procrustes before upload, alternating factors across rounds with a soft interpolation weight. It is the strongest competitor because it targets the same symmetry with essentially no communication overhead. It reports 87.34% on MNLI (RoBERTa-Large, 50 clients) and 40.88% HumanEval pass@1 (Llama 3-8B), against 82.44% for RoLoRA.

Strongest competitor under heterogeneous ranks: FedPA-LoRA. It never overwrites client factors and adds a product-space penalty toward the global update. It reports 92.26% average GLUE accuracy versus 85.44% for HetLoRA, at roughly 34% extra wall-clock per round on Llama 3-8B. We also run the alternating-factor method RoLoRA from Robust Federated Finetuning of Foundation Models via Alternating Minimization of LoRA. For vision we compare against FedLoRU, which reports 57.96% versus 53.79% for FedLoRA on CIFAR-100 with 100 clients. For GSM8K we found no reported result on Lacuna for any of these methods, so we will run all of them ourselves.

Fairness protocol. Every method uses the same backbone, client count, Dirichlet partition seeds, LoRA rank, rounds, local steps and optimizer. Each method gets the same tuning grid: six learning rates crossed with that method's own key hyperparameter. That hyperparameter is FedRot's soft-rotation weight, FedPA's penalty weight, or our initialization scale $\alpha$, so our extra knob is charged against our budget. We report mean and spread over three seeds. We will reproduce the published FedRot-LoRA and FedPA-LoRA numbers within about one point before trusting our own comparisons.

What counts as success. Our method costs clients nothing. Matching FedRot-LoRA within half a point on MNLI and HumanEval at equal communication, and matching FedPA-LoRA under heterogeneous ranks without its 34% overhead, would already support the claim. Exceeding FedRot-LoRA at high rank, such as rank 24 where RoLoRA is reported to collapse, or in early rounds where FedRot's reference is noisy, would show that the balance component matters in practice.

Ablations. These isolate the key ingredient:

  • balanced initialization alone, without server rebalancing;
  • server rebalancing alone, from standard LoRA initialization;
  • small but unbalanced initialization;
  • our method plus FedRot's client-side Procrustes, to test whether rotation alignment still adds anything once balance is enforced.

Why not just…?

Why not just use FedRot-LoRA's Procrustes alignment, which is already cheap? This is the most serious alternative, and we treat it as a real threat. Procrustes searches only over orthogonal rotations. The factor freedom is the full group of invertible matrices, so any scale or shear mismatch between clients passes through it untouched. Standard LoRA initialization creates exactly such mismatch through its unbalanced conserved quantity. The gap decomposition settles this. If the balance part is small under standard initialization, our extra mechanism is unnecessary and the honest result is "FedRot suffices." In that case the contribution narrows to explaining when and why misalignment arises. If the balance part is large, the our-method-plus-Procrustes ablation should beat Procrustes alone.

Why not just average dense products on the server and broadcast a truncated SVD? FedPA-LoRA documents that this overwrites client factors and optimizer state. The resulting initialization mismatch degrades training. Our method also re-broadcasts factors, so the same objection applies to us. Our answer is that the re-factorization targets the balanced form, with the orientation closest to the clients' own factors. The restart should therefore barely move a client that was already balanced and aligned. We will measure the relative initialization error per round, the quantity FedPA-LoRA reports as zero for itself. If ours is not small, we will adopt FedPA's keep-local-factors design and replace only its penalty.

Why not just freeze $A$ or alternate factors, which makes averaging exact? With a frozen shared $A$, factor averaging is exactly product averaging, so this alternative is real. The cost is capacity. The RoLoRA paper reports that freezing $A$ loses accuracy as budgets shrink and heterogeneity grows. FedRot-LoRA reports RoLoRA collapsing at rank 24. Our rank sweep from 4 to 32 will show whether training both factors from a balanced start keeps exactness without that collapse.

Why not just use magnitude-based Top-$k$ sparsification instead of low-rank updates? Discrepancy-aware calibration shows that magnitude selection is blind to how updates change model outputs after multi-step local training. It reports 21.03% versus 29.62% on CIFAR-10 with ViT-tiny at a 0.01 compression ratio. We include Top-$k$ at matched bytes as a sanity reference, not as a main competitor.

Proposed Experiments

Synthetic matrix sensing. We sweep the initialization scale $\alpha$ over four orders of magnitude, client heterogeneity measured as principal angle, local steps $T$, and search rank from 3 to 100. The prediction is that $\|\Delta\|_F$ scales linearly with $\alpha$ and with subspace angle, and stays flat in search rank. Under standard initialization it should not.

Language understanding. We use MNLI, SST-2, QNLI, QQP and RTE from GLUE with RoBERTa-Large and 50 clients, under Dirichlet label splits and both homogeneous and heterogeneous ranks. This matches the FedRot-LoRA and FedPA-LoRA settings.

Generative tasks. We fine-tune Llama 3-8B and evaluate on GSM8K and HumanEval pass@1. We also track invariant drift under AdamW, to quantify how much the server rebalancing is doing.

Vision. We run CIFAR-100 with 100 clients against FedLoRU, which tests whether the "rank paradox" filtering effect FedLoRU reports adds to the alignment effect.

Metrics. We report task accuracy, bytes to reach a target accuracy, client wall-clock per round, the two-part aggregation gap, and initialization error.

Decisive check. Across runs, the rotation part of the gap under our method should be predicted by measured client subspace disagreement. If it is not, the proposed mechanism is wrong even if accuracy improves. If the balance part is negligible under standard initialization, the central claim fails.