Spending the Privacy Budget Where Heterogeneity Needs It: Online Round Planning for Private Personalized Federated LoRA

When many hospitals or phones fine-tune one shared model without pooling their data (federated learning), each device sends model updates to a server. To give a formal privacy guarantee, each device adds random noise to its update. This is differential privacy: a bound on how much any one person's data can change what leaves the device. The noise adds up over training rounds. Under a fixed total privacy budget, every extra round forces more noise into every round, so accuracy first rises and then falls. Prior work calls this a U-shaped error curve with a best stopping round $T^*$ (Personalized and Private Peer-to-Peer Machine Learning). The only planning tool for choosing $T$ in advance assumes every client's data comes from the same distribution (Proactive DP). Real clients hold very different data, which pulls their updates in different directions and slows convergence. That pushes $T^*$ later just as the privacy budget pushes it earlier.

We propose to choose, while training runs, two things: when to stop, and which part of a low-rank adapter is shared versus kept on the device. A low-rank adapter (LoRA) is a small trainable add-on to a frozen pretrained model. Both decisions are made only from the noisy updates the server already receives. By the post-processing property of differential privacy, any computation on already-privatized outputs costs no additional privacy, so this planning is free. In two sentences: heterogeneity decides how many rounds you need, and the privacy budget decides how many you can afford. We estimate the first from already-noised updates, at zero extra privacy cost, and cut how much must be shared by keeping the most client-specific adapter pieces local.

Core Claim

One design variable ties all four concerns together: the size and identity of the shared, transmitted part of the adapter. Only the shared part is noised, so it alone accumulates privacy noise, pays communication cost, suffers client drift when averaged, and gives an attacker gradients to invert. Everything kept local is noise-free, costs no bandwidth, and never leaves the device.

A simple error model makes the trade-off explicit. Let $d_s$ be the number of transmitted adapter parameters and $\zeta_s$ the dissimilarity of client updates on that shared part. Let $K$ be the number of clients per round and $\epsilon$ the total budget. Then

$$\text{err}(T, d_s) \approx \frac{C(\zeta_s)}{T} + b(\zeta_s) + c\,\frac{T\, d_s}{K\,\epsilon^2}.$$

Here $C(\zeta_s)$ is an optimization constant that grows with heterogeneity, and $b(\zeta_s)$ is the bias from averaging drifting clients. The last term is accumulated noise. It follows because under Gaussian-mechanism composition the per-round noise standard deviation scales like $\sqrt{T}/\epsilon$, and its squared norm scales with dimension. Balancing the first and last terms gives $T^* \propto \epsilon\sqrt{K\,C(\zeta_s)/d_s}$.

Two predictions follow, and both are testable. First, moving the most client-specific adapter components off the wire lowers $d_s$ and $\zeta_s$ together. That should both deepen the minimum of the U-curve and move it. Second, because the noise level is public, the server can estimate $\zeta_s$ and the signal-to-noise ratio from per-client noisy updates without bias. It can then track $T^*$ online instead of fixing it in advance.

The new content is not "add personalization to DP-LoRA." It is a zero-privacy-cost controller over stopping and sharing, validated against the heterogeneity regimes where fixed plans fail.

Relation to Prior Work

Three recent methods fix the noise problem specific to LoRA. Naive private LoRA collapses because noise added to both factor matrices multiplies into a noise-times-noise term, and because averaging products of matrices is not the product of averages.

Improving LoRA in Privacy-preserving Federated Learning freezes the down-projection $A$ and transmits only $B$. DEeR alternates $A$ and $B$ across global rounds and adds a noise regulator. Rethinking LoRA for Privacy-Preserving Federated Learning in Large Models (LA-LoRA) alternates them locally within a round and low-pass filters gradients. We reuse FFA-LoRA's linearization as the substrate. None of the three chooses $T$ adaptively, and none keeps any adapter parameters private. LA-LoRA's authors explicitly list extreme data silos as open, and DEeR's cross-round alternation spends more rounds, which is exactly what our planner prices.

The personalization literature supplies the local-versus-shared split. FedRep keeps classifier heads local. FedBN keeps normalization layers local. Ditto regularizes a personal model toward a global one. All three are non-private. Ditto names the interaction of privacy noise with personalization as future work, which motivates us directly. Proactive DP supplies the planning idea but assumes independent, identically distributed data. We replace its a-priori plan with an online one that measures heterogeneity. Recycled ADMM also exploits post-processing, by reusing past updates as extra optimization steps in convex problems. We use post-processing for control decisions instead, in non-convex adapter training.

Gradient leakage enters as the threat model that constrains the design. MineGrad shows that a malicious server can poison the frozen backbone and craft adapter down-projections so that LoRA gradients reveal 85.9% of CIFAR-100 image patches. This is a negative result for FFA-LoRA as usually deployed, because the server picks the frozen $A$. We therefore require every arm to derive $A$ from a public, pre-committed seed and to verify the backbone checksum. We then measure whether any leakage remains under both the passive and active attacks. Evaluating Gradient Inversion Attacks and Defenses shows that inversion depends on normalization statistics, which strengthens the case for keeping normalization local.

Research Program

We would build a client-level private federated LoRA system on the DEeR codebase (CUHK-AIM-Group/DEeR). In this system, each client clips and noises only its shared $B$ blocks. Local heads, normalization parameters, and a chosen set of "personal" $B$ blocks never leave the device.

The server runs a controller on the noisy per-client updates it receives. It maintains debiased running estimates of the aggregate update norm, used for signal-to-noise, and of the across-client dissimilarity per adapter block. The debiasing subtracts the known noise variance.

From these estimates the controller does two things. It fits the one-dimensional error model above to predict the remaining gain from another round, and it stops when that gain turns negative. It also reassigns the blocks whose estimated dissimilarity is highest from shared to personal. Reassignment only shrinks the shared set, never grows it, so the privacy accounting for already-spent rounds is unaffected.

Because the total round count is no longer fixed in advance, we would use a privacy accountant that allows adaptive stopping. We would budget for a maximum round count $T_{\max}$, and report both the realized and the worst-case $\epsilon$ honestly.

The analysis has two parts. The first proves, for a linear representation model in the style of FedRep, that the planner's stopping rule tracks $T^*$ up to estimation error in $\zeta_s$. The second characterizes when moving a block to the personal set lowers total error, namely when its drift contribution exceeds its share of the optimization signal.

Baselines

The naive reference is vanilla private LoRA with noise on both $A$ and $B$ and a fixed $T$. Improving LoRA in Privacy-preserving Federated Learning reports it collapsing to 33.80% on MNLI at $\epsilon=1$. DEeR reports it at 42.15% on OCT-C8 at $\epsilon=0.1$. Any proposal that does not clear this bar is uninteresting.

The strongest existing competitors are LA-LoRA and DEeR.

  • LA-LoRA reports 60.68% test accuracy on Tiny-ImageNet with Swin-B at $\epsilon=1$ (Rethinking LoRA).
  • DEeR reports 84.28% on OCT-C8 at $\epsilon=0.1$ (DEeR).
  • FFA-LoRA reports 75.05% on MNLI at $\epsilon=1$.

These methods solve noise amplification but use fixed round counts. We would rerun all three on our own partitions, since the file does not state their partition schemes. Each would get the same backbone, clients, per-client data, compute budget, and the same grid of rounds, learning rates, and clipping norms. Hyperparameters would be selected on a held-out non-private proxy split, identically for all methods.

Proactive DP's a-priori plan, applied to FFA-LoRA, is the direct competitor on round planning. We also report an oracle that picks the best $T$ in hindsight. This is an upper bound no private method can legitimately reach.

For personalization reference points, we include FedRep and Ditto with private shared updates at a fixed $T$. FedRep's non-private 87.7% on CIFAR-10 with 2 classes per client is the ceiling for that partition.

Two ablations isolate the key ingredients. The first is the same system with a static split, fixed at initialization, plus the online stopping rule. The second is the online split with a fixed, tuned $T$.

Beating the field means two things. At matched $\epsilon$ and bandwidth, the full system must exceed LA-LoRA, DEeR, and FFA-LoRA on mean and 10th-percentile client accuracy under class-disjoint partitions. It must also come within the oracle's accuracy range on most seeds, where Proactive DP does not.

Why not just…?

Why not just grid-search $T$ for FFA-LoRA or LA-LoRA? Tuning $T$ on private client validation data spends budget that is rarely accounted for, and a tuned $T$ does not transfer across heterogeneity levels. This is a real threat if the best $T$ turns out to be stable across partitions. The oracle-$T$ comparison, run on Dirichlet partitions with varying concentration, settles it. If one fixed $T$ stays near-oracle across all partitions, the stopping controller adds nothing, and we would report that.

Why not just bolt a FedRep-style local head onto FFA-LoRA with a fixed split? This is the most dangerous alternative, because it captures much of the drift reduction for free. The static-split ablation measures it directly. If static and adaptive splits match, the contribution narrows to the zero-cost stopping rule. We expect the gap to appear only in class-disjoint and multi-domain regimes, where which blocks are client-specific is not known in advance.

Why not just verify the backbone and fix $A$ by public seed, and skip DP? These checks probably defeat PEFTLeak as published, because the attack depends on server-chosen frozen weights. That is why we adopt them everywhere. They give no guarantee against passive, multi-round optimization inversion or future attacks, though. SHE-LoRA's cryptographic route (SHE-LoRA) likewise trusts a non-colluding server. We will report a passive inversion attack's reconstruction quality with hygiene alone versus hygiene plus our noise. If hygiene alone already drives passive reconstruction to chance, the privacy motivation rests on the formal guarantee rather than empirical leakage, and we will say so.

Why not just use Proactive DP's closed-form plan? It predicts $T$ from $\epsilon$ and batch size alone, with no heterogeneity term. Our first decisive check tests exactly whether that omission shifts the optimum.

Proposed Experiments

We evaluate vision on CIFAR-100 and Tiny-ImageNet with ViT/Swin LoRA, and medical cross-silo data on OCT-C8 and Kvasir-v2. The vision partitions range from IID through Dirichlet splits to FedRep-style few-classes-per-client. For language we use MNLI split by genre as a feature-shift partition. We test $\epsilon \in \{0.1, 1, 8\}$.

Metrics are mean and 10th-percentile client accuracy, following LEAF's percentile reporting (LEAF). We also report total uplink bytes, realized versus worst-case $\epsilon$, and the distance of the chosen $T$ from the oracle $T^*$.

The first decisive check tests whether measured heterogeneity moves $T^*$ in the direction and roughly the magnitude our model predicts. We sweep Dirichlet concentration at fixed $\epsilon$. We accept the model if the oracle $T^*$ moves with the controller's estimate and away from Proactive DP's constant plan.

The second check is whether the full system beats LA-LoRA and DEeR on 10th-percentile accuracy under class-disjoint partitions at matched bytes. The ablations would say which of the two ingredients carries the gain.

The third check covers leakage. We run PEFTLeak, reported at 85.9% patch recovery on CIFAR-100 and 81% on Tiny-ImageNet at batch 32 (MineGrad), against unprotected LoRA, hygiene only, and hygiene plus noise. We also run a passive optimization attack, scored by LPIPS against the 0.45 undefended and 0.73 GradPrune-plus-InstaHide references (Evaluating Gradient Inversion). This tells us whether stopping earlier and sharing less also measurably lowers what an attacker recovers over many rounds.