In federated learning, many devices train one shared model without pooling their data. Each device sends model updates, not raw examples. Two problems make this harder than it sounds. First, the updates themselves can be inverted to recover training images, so the devices need a formal privacy mechanism. The standard one is differentially private stochastic gradient descent (DP-SGD): it caps the length of each example's gradient at a threshold $C$ (clipping) and adds Gaussian noise, so that no single example measurably changes what is sent. Second, devices hold different data, which is called statistical heterogeneity. A device whose data looks unusual pulls the model in its own direction, a pull known as client drift. Those unusual directions produce long, atypical gradients, and clipping cuts them hardest.
The proposal is simple. Before clipping, split each per-example gradient into two parts. One part lies in a low-dimensional "consensus" subspace that is estimated only from updates the server has already released. The other part is the leftover, which carries the client-specific drift. Only the consensus part is clipped, noised, and sent. The leftover is never discarded: it updates a private offset that stays on the device and personalizes that client's model. Our claim is that this one change helps with three things at once. Clipping stops taxing unusual clients, because their drift is removed before the length cap applies. Noise is added in $k$ dimensions instead of $d$. And what leaves the device lies in a subspace shared by all clients, which should make individual examples harder to separate by gradient inversion. The formal $(\epsilon,\delta)$ guarantee is exactly that of DP-SGD on the released message.
Core Claim
Write $W_t$ for the shared parameters at round $t$. Let $U_t \in \mathbb{R}^{d\times k}$ be an orthonormal basis computed from the top singular directions of the last few noisy global updates, topped up with a few random directions so the subspace can keep exploring. Because $U_t$ depends only on outputs that were already released, using it costs no extra privacy. For a local batch $B$ with per-example gradients $g_i$, client $j$ sends
$$ m_j = \frac{1}{|B|}\Big(\sum_{i\in B}\mathrm{clip}_C\big(U_t^\top g_i\big) + \mathcal{N}(0,\sigma^2C^2 I_k)\Big), $$
and privately updates its offset $\delta_j$ with the never-released residual,
$$ \delta_j \leftarrow \delta_j - \eta\,\frac{1}{|B|}\sum_{i\in B}(I - U_tU_t^\top)\,g_i . $$
Here $\mathrm{clip}_C$ rescales a vector to norm at most $C$, $\sigma$ is the noise multiplier, and $\eta$ is the local step size. The server aggregates the $m_j$ and applies $U_t \bar m$ to $W_t$. Client $j$ predicts with $W_t + \delta_j$.
Sensitivity is bounded by $C$ whatever $U_t$ is. So the guarantee holds even if a malicious server chooses $U_t$ adversarially. That matters because MineGrad shows active servers can shape the parameters clients receive so that updates leak data.
We make three testable predictions.
- Clipping tax. Clipping bias concentrates on high-drift clients, and it should shrink once drift is routed to $\delta_j$.
- Leakage profile. At fixed $\epsilon$, examples from atypical clients are empirically more reconstructable under standard DP-SGD. Their gradients are less entangled with the rest of the batch, which is the condition DCS² identifies as enabling inversion. This gap should close under consensus projection.
- Accuracy. Personalized accuracy at $\epsilon=4$ should exceed the best reported differentially private personalized method.
We do not claim to beat worst-case limits. Fundamental Limitations of Favorable Privacy-Utility Guarantees for DP-SGD constrains that, and we accept the constraint. Our gains are in typical-case utility and in empirical leakage.
Relation to Prior Work
Several papers motivate the gap without closing it.
- Disparate Impact in Differential Privacy from Gradient Misalignment shows that clipping changes the direction of the average gradient and hurts minority groups. It repairs this centrally with a global rescaling (DPSGD-Global-Adapt). We treat client drift as the federated version of that misalignment. We also fix it by removing the drift before clipping, rather than by rescaling the whole batch after it.
- Fu et al. and Beyond Aggregation document drift, but not how it interacts with clipping.
Other papers supply the method's ingredients.
- Subspace perturbation (Yang et al., Ács et al.) shows that noise confined to a small subspace distorts training less. Their subspaces are fixed architectural choices. Ours tracks the released consensus and keeps the complement as personalization instead of freezing it.
- Ditto and pFedMe supply the idea of a private offset from a global model. They pay no attention to clipping.
The closest competitors decide what to share by a static or scheduled rule.
- FedRep shares the body and keeps the classifier head local, with no privacy mechanism.
- Aggregate or Not? finds that privatizing encoder layers can beat privatizing heads. We read that contradiction as evidence that drift does not respect layer boundaries. This motivates splitting by direction within parameters rather than by layer.
- FedDPA keeps high-Fisher-information parameters local and adds a loss term that keeps updates near the clipping threshold. Its split is coordinate-wise and is decided before local training. Ours is directional, updated from released outputs, and applied per example before clipping.
- SDFLoRA applies noise only to a shared LoRA module.
None of these three measures gradient-inversion leakage.
Finally, Group privacy for personalized federated learning defends against gradient inversion under heterogeneity, but it uses metric privacy, which loosens in high dimensions. We stay within standard $(\epsilon,\delta)$-DP.
Research Program
The work has four parts.
- Diagnostic study. Under DP-FedAvg, we measure per client:
- the fraction of per-example gradients clipped;
- the cosine between the clipped and unclipped mean update;
- reconstruction quality under gradient inversion.
We plot each against a heterogeneity score: the distance between the client's label distribution and the global label distribution. This tests the leakage prediction directly. If atypical clients are not more reconstructable at fixed $\epsilon$, the leakage prediction fails. The proposal would then rest on the utility claim alone, and we would say so.
- The method. We build it with $k$ chosen per round, using a small private count of how much gradient energy falls outside the subspace, budgeted as in DPSGD-Global-Adapt. The subspace is refreshed from a sliding window of released updates.
- A short convergence analysis. It separates clipping bias restricted to the subspace from the residual left for personalization, so the error terms can be compared directly with DP-FedAvg.
- Parameter-efficient extension. We port the method to a DP-LoRA setting to see whether it helps where the file reports the largest gaps. There, the projection is applied to the gradient of the LoRA $B$ matrix under alternating updates, following LA-LoRA.
Baselines
All baselines share the same data partitions, backbone, number of rounds, and client sampling rate. They are calibrated to identical $(\epsilon,\delta)$ with the same privacy accountant, and each gets the same hyperparameter budget of 20 configurations over clipping norm, learning rate, and method-specific knobs.
- Naive reference: DP-FedAvg. Standard per-example clipping and Gaussian noise on a single global model.
- Strongest existing method: FedDPA. It is the only cited method that is personalized, differentially private, and reports numbers on our benchmarks at $\epsilon=4$:
- FEMNIST: 77.27% accuracy;
- SVHN: 62.63% accuracy;
- CIFAR-10 Dirichlet $\alpha=1$: 49.75% accuracy.
We rerun it under our shared protocol rather than copying these numbers. Beating it means higher mean personalized accuracy at equal $\epsilon$ on at least two of these three benchmarks, with no loss in worst-decile client accuracy.
- FedRep with DP-SGD applied to the shared body. This is the static layer split plus formal privacy. The FedRep paper reports 87.7% on 2-class CIFAR-10 without privacy, which serves as a non-private ceiling.
- DP-Ditto. A private offset with a full-dimensional noised global update. This isolates what projection adds beyond personalization.
- DPSGD-Global-Adapt ported to federated clients.
- Ablations that isolate the key ingredient.
- Random subspace. Replaces the consensus subspace with a random $k$-dimensional one of equal size. This tests whether the subspace's alignment with consensus matters, or only its size.
- Discard residual. Throws the residual away instead of feeding $\delta_j$. This tests whether personalization matters, or only the reduced noise.
- Leakage baselines.
- DCS², which reports SSIM 0.15 on CIFAR-10;
- mixed quantization, which reports DLG PSNR 8.1 on MNIST.
These are reference points without formal guarantees.
Why not just…?
Why not just keep the classifier head local and apply DP-SGD to the shared body? This is the most serious threat. If most drift lives in the head, a static split already removes it before clipping. The Aggregate-or-Not findings suggest drift also lives in the encoder, which a head-only split cannot catch. Still, this baseline could match us on label-shift benchmarks. The decisive comparison is on FEMNIST, whose writer-level style differences are covariate shift, and on CIFAR-10 with Dirichlet $\alpha=0.1$. There we report the fraction of clipped gradient energy inside the shared body for both methods. If FedRep with DP matches us on accuracy and clipping statistics, the directional split adds nothing and we will report that.
Why not just use adaptive or group-aware clipping? DPSGD-Global-Adapt rescales a batch uniformly. That corrects the length of the average gradient but not its direction when a whole client drifts. It has also only been evaluated centrally, reaching 65.5% vs. 26.3% for standard DP-SGD on the 1% MNIST minority class. We port it directly, and measure the angle between the clipped and unclipped mean update for high-drift clients under each method.
Why not just add noise in a smaller subspace, such as LoRA or random projections, without personalization? Smaller noise dimension is part of our gain, and the random-subspace and discard-residual ablations separate it out. If the random subspace matches the consensus subspace, the "consensus" ingredient is unnecessary and the method reduces to known subspace perturbation.
Why not just use heuristic defenses like decoys or quantization? They report strong reconstruction numbers, but they give no formal bound. MineGrad reports active attacks still working against quantization and low-level noise, recovering 82.8% of CIFAR-10 patches at batch size 32. We include them only to calibrate what a given reconstruction score means.
Proposed Experiments
Utility. We use FEMNIST (natural writer partition), CIFAR-10 (Dirichlet $\alpha \in \{0.1, 1\}$ and 2 classes per client), SVHN, and CIFAR-100 for severe label skew. We sweep $\epsilon \in \{1,2,4,8\}$ and report:
- mean and worst-decile personalized accuracy;
- the gap in privacy cost between low-drift and high-drift client terciles, following the privacy-cost-gap metric of the disparate-impact paper.
Leakage. We run three kinds of gradient inversion attack against released updates at batch sizes 8, 32, and 64:
- DLG-style optimization with cosine distance;
- a generative-prior attack following GIFD;
- a MineGrad-style active server that chooses $U_t$ and backbone weights adversarially.
We report PSNR, SSIM, and patch-recovery rate, stratified by the client heterogeneity score.
Decisive checks. Three results would support the proposal together:
- under DP-FedAvg, reconstruction quality rises with heterogeneity at fixed $\epsilon$, and our method flattens that slope;
- the consensus subspace beats the random subspace of equal $k$;
- our method beats FedDPA and FedRep with DP on FEMNIST and CIFAR-10 with $\alpha=0.1$ at $\epsilon=4$.
Secondary transfer test. Tiny-ImageNet with Swin-B and LoRA at $\epsilon=1$, against LA-LoRA's reported 60.68% accuracy.
References
- MineGrad: Gradient Inversion Attacks on LoRA Fine-Tuning
- Concealing Sensitive Samples against Gradient Leakage in Federated Learning
- Fundamental Limitations of Favorable Privacy-Utility Guarantees for DP-SGD
- Disparate Impact in Differential Privacy from Gradient Misalignment
- FedDC: Federated Learning with Non-IID Data via Local Drift Decoupling and Correction
- Beyond Aggregation: Guiding Clients in Heterogeneous Federated Learning
- Differentially Private Adapters for Parameter Efficient Acoustic Modeling
- Constrained differentially private federated learning for low-bandwidth devices
- Ditto: Fair and Robust Federated Learning Through Personalization
- Personalized Federated Learning with Moreau Envelopes
- Exploiting Shared Representations for Personalized Federated Learning
- Aggregate or Not? Exploring Where to Privatize in DNN Based Federated Learning Under Different Non-IID Scenes
- Dynamic Personalized Federated Learning with Adaptive Differential Privacy
- SDFLoRA: Selective Decoupled Federated LoRA for Privacy-preserving Fine-tuning with Heterogeneous Clients
- Group Privacy for Personalized Federated Learning
- Rethinking LoRA for Privacy-Preserving Federated Learning in Large Models
- Mixed Quantization Enabled Federated Learning to Tackle Gradient Inversion Attacks
- GIFD: A Generative Gradient Inversion Method with Feature Domain Optimization