One Shared Projection for Bandwidth, Secure Masking, and Privacy Noise in Federated Learning

In federated learning, many devices train one model together by sending weight updates to a server instead of sending their raw data. Three costs pull against each other in this setting. Updates are large, so they are usually compressed. Updates leak private data, so they are protected with cryptographic masking (secure aggregation, which lets the server see only the sum of client updates) and with differential privacy noise. And every compressor, mask, and noise step adds on-device computation, which can cancel out the network time it was meant to save. Today these three problems are handled by separate components that often conflict. Top-$k$ sparsification gives each client a different set of coordinates, which breaks secure aggregation. Low-rank methods such as PowerSGD reduce noise well but depend on expensive matrix factorizations. Quantizers that work with secure aggregation keep the full dimension $d$, so masking cost still grows with model size.

We propose that every client apply the same pseudo-random linear projection, chosen fresh each round from a public seed, and that clipping, secure aggregation, and differential privacy noise all be applied to the resulting $k$-dimensional sketch, with $k \ll d$. Each client keeps an error-feedback memory of the part of its update that the projection discarded. A single step then does three jobs. It gives all clients identical coordinates, so masks cancel without any negotiation. It makes masking and upload cost scale with $k$ instead of $d$. And it confines privacy noise to a $k$-dimensional subspace, so much less noise reaches the model at the same privacy level. We claim that a structured version of this projection is cheap enough to reduce end-to-end time to target accuracy on edge hardware, and that under client-level differential privacy it beats both uncompressed private training and the strongest secure quantized aggregation at equal privacy and equal bytes.

Core Claim and Why It Is Not Already Done

The closest existing systems each cover two of the three pieces. ScionFL combines secure multi-party aggregation with 1-bit quantization after a randomized Hadamard transform. This is the proposal's strongest competitor, but it keeps all $d$ coordinates, adds no differential privacy, and assumes a semi-honest server committee. THC uses the same Hadamard-plus-quantization idea so that servers and switches can add compressed tensors directly. It targets throughput in data centers and does not analyze privacy. CESAR makes secure aggregation work with sparsification through multi-hop agreement on shared coordinates. That costs extra communication rounds and provides neither dropout tolerance nor error feedback.

On the Interaction Between Differential Privacy and Gradient Compression motivates the noise argument. It shows that PowerSGD's low-rank compression filters out privacy noise, reporting 10.4% higher accuracy on Surnames at $\epsilon = 50.92$. However, that study is centralized, relies on factorizations that SEPARATE identifies as a wall-clock bottleneck, and reports large models failing to converge. SEPARATE supplies our method. It replaces the factorization with a shared random Gaussian projection plus moving-average error feedback, but it was studied only in non-private, centralized training. No paper in this file places a shared-seed projection underneath secure aggregation and distributed differential privacy and then accounts for the full client-side time budget.

The quantitative reason to expect a gain is simple. Let $P$ be a $k \times d$ projection with orthonormal rows, $g$ a client's update of size $d$, $C$ the clipping norm that would be used in full dimension, and $z$ the noise multiplier set by the privacy budget.

For an isotropic update, $\|Pg\| \approx \sqrt{k/d}\,\|g\|$. The clipping threshold in sketch space can therefore be $\sqrt{k/d}\,C$ while clipping the same fraction of clients. Decoding with $P^\top$ keeps a $k/d$ share of the signal energy and adds noise energy $k \cdot z^2 (k/d) C^2$. The per-round signal-to-noise ratio then compares as

$$\frac{\mathrm{SNR}_{\text{projected}}}{\mathrm{SNR}_{\text{full}}} = \frac{(k/d)\|g\|^2 \,/\, (z^2 k^2 C^2/d)}{\|g\|^2 \,/\, (z^2 d\, C^2)} = \frac{d}{k}.$$

The catch is that only a $k/d$ share of the signal gets through in each round. The rest has to be delivered later by error feedback. Whether that delayed signal turns into accuracy is the central empirical question.

Two further points follow from this derivation. First, rescaling the decoded update by $d/k$ to make it unbiased would magnify the noise and remove the gain. The design therefore uses a deliberately biased decoder together with error feedback. Second, error feedback does not weaken the privacy guarantee under client-level differential privacy, provided clipping is applied to $P(g+e)$ rather than to $g$ alone, where $e$ is the client's error-feedback memory. Each round's released contribution then has norm at most the clip bound, whatever private state produced it, so standard adaptive composition applies. This addresses the concern in the research file's Gap 4 that moving-average error feedback might erode privacy across rounds.

Research Program

We will build a client pipeline with five stages. The first is a projection, reseeded every round from a public seed. It uses a subsampled randomized Hadamard transform for vector-shaped layers, costing $O(d \log d)$ and following THC's transform, and a SEPARATE-style right-multiplication by a Gaussian matrix for matrix-shaped layers. The second is clipping in sketch space. The third is integer quantization of the sketch. The fourth is a distributed discrete Gaussian noise share, sized so that the sum reaches the target client-level $\epsilon$ once enough clients survive. The fifth is secure aggregation of the $k$-dimensional integer vectors using LightSecAgg, chosen because its one-shot mask recovery tolerates dropouts, which CESAR lacks.

The seed is public, so privacy never depends on keeping it secret. This is a deliberate contrast with Mixed Quantization, whose protection rests on keeping per-client scaling keys hidden.

Following the time decomposition that motivates Direction 1, we will instrument every stage: encoding, masking, transfer, server-side unmasking, and decoding. This lets us report, for each hardware and network setting, whether compression actually pays for itself.

The analysis has two parts. The first is a convergence argument for biased projection with error feedback under clipped, noised aggregation, extending the SEPARATE analysis to the private setting. The second measures the $d/k$ gain in signal-to-noise ratio on real, anisotropic gradients, whose energy is concentrated in few directions, since the isotropic assumption above will not hold exactly.

We will also run a separate audit of what the projection protects by itself. We will run Generative Gradient Leakage and Deep Leakage from Gradients on individual projected updates with secure aggregation and noise removed. Because the server knows $P$, we expect projection alone to be a weak defense. Measuring this directly keeps the proposal from claiming privacy through obscurity.

Baselines

The naive reference points are plain uncompressed federated averaging with no privacy, which sets the accuracy ceiling, and uncompressed client-level private federated averaging over LightSecAgg, which sets the privacy-matched floor. LightSecAgg reports a 12.7× speedup over the original secure aggregation protocol. For this private floor we found no reported accuracy on Lacuna under our exact settings, so we will measure it ourselves.

The strongest existing competitor is ScionFL-style secure 1-bit Hadamard quantization. It is the only method in the file that already makes compression compatible with secure aggregation. ScionFL reports MNIST validation accuracy dropping only from 99.04% to 98.71% after 1,000 rounds, and a 17.2× reduction in inter-server communication with 500 clients. The file gives no reported number for its CIFAR-10 ResNet-9 runs. We will add the same distributed noise to ScionFL so that both methods are compared at equal $\epsilon$. We will then compare at equal uploaded bytes per client per round. ScionFL is fixed at about $d$ bits; our method is set to $k$ times the integer width.

The comparison will be fair in three ways. Both methods run on MNIST and CIFAR-10 with ResNet-9 and use the same client splits: an IID split and a Dirichlet non-IID split. Both get the same hyperparameter search budget over learning rate and clipping norm. Both are timed on the same hardware.

We will also include CESAR-style top-$k$ sparsification with intersection masking, where CESAR reports accuracy within 0.5% of unmasked training and 11% data overhead. We will also include federated private PowerSGD with the Denoise error feedback from the differential privacy and compression study. These two represent the sparsification and factorization alternatives.

For language models, the baseline is private federated LoRA on GPT-Neo 1.3B with CNN/DailyMail. ERIS reports reducing the upload for this model from 5.2GB to 46.8MB, which gives a byte-level reference point.

Beating a baseline means two things. At equal $\epsilon$ and equal bytes, we need higher final accuracy, or equal accuracy in less wall-clock time, over at least three seeds. On the edge device, measured encode-plus-mask time must stay below the transfer time saved.

The ablations isolate the key ingredient. They remove error feedback, clip in full dimension instead of sketch space, use the unbiased rescaled decoder, reuse one fixed projection across rounds, and replace the structured transform with a dense Gaussian matrix. Removing error feedback, or using the unbiased decoder, should eliminate most of the gain if our account of the mechanism is right.

Why not just…?

Why not just add local differential privacy noise to uncompressed updates? The file already reports this option doing poorly. Differential privacy baselines in the mixed quantization study fell to 78–85% MNIST accuracy, against 96.67% for quantization. The differential privacy and compression study reports ResNet-18 and BERT stuck at random-guess accuracy under high-dimensional noise. Our uncompressed private floor reproduces this comparison at matched $\epsilon$. The $d/k$ argument is precisely a claim that this baseline wastes its noise budget.

Why not just use secure 1-bit quantization, as ScionFL or THC do? This is the real threat. It is simple, already compatible with secure aggregation, and loses little accuracy on MNIST. Its weaknesses are that bytes and masking cost stay proportional to $d$, and that its noise, once added, lives in all $d$ dimensions. If ScionFL with distributed noise matches our accuracy at equal $\epsilon$ and equal bytes on non-IID CIFAR-10 and the LoRA task, the projection's added complexity is not justified and we will report that. The equal-bytes sweep settles the question. At high byte budgets we expect ScionFL to win. We claim an advantage only where $k$ falls well below $d/32$.

Why not just sparsify aggressively with top-$k$? Client-specific coordinates make secure aggregation impossible without CESAR's extra coordination rounds. Evaluating Gradient Inversion Attacks and Defenses reports roughly a 10% accuracy loss from pruning more than 99% of gradients. GGL reconstructs inputs from only 10% of gradient entries. The CESAR baseline measures how much of this coordination cost remains in practice.

Why not just rely on LoRA, which already shrinks the update? MineGrad shows that small updates do not provide privacy: it recovers 85.9% of CIFAR-100 patches at batch size 32. LoRA reduces bytes but offers no noise-filtering effect. Our language model experiment applies the projection on top of LoRA updates, so any gain beyond LoRA alone shows up directly. If that gain is negligible, the projection's benefit is confined to full-model training.

Proposed Experiments

Three decisive checks decide the proposal. The first is an accuracy-versus-$\epsilon$ curve at matched bytes on non-IID CIFAR-10 with ResNet-9, together with the measured per-round signal-to-noise ratio plotted against $d/k$. If the measured gain falls far below $d/k$ on real gradients, and accuracy does not improve over ScionFL with noise, the core claim fails.

The second is a full wall-clock breakdown on a Raspberry Pi 4 over a constrained uplink, the device used by EdgeDetect, with the same pipeline also timed on a GPU over a fast link. We expect compression to pay off on the edge device and to be neutral or harmful on the fast link, as the direction overview predicts for high-speed interconnects. We will report both results.

The third is a privacy audit on CIFAR-10 and CelebA covering three conditions: GGL and Deep Leakage from Gradients against raw projected updates, against the full pipeline, and against a MineGrad-style active server that crafts the weights it sends to clients. Results are reported as PSNR and LPIPS and compared with the published undefended reconstruction PSNRs: 31.7 dB in EdgeDetect and 55.3 dB in the mixed quantization study. Under the active attack, the formal guarantee still holds because clipping and noise happen on the client. The audit shows how much reconstruction quality an attacker actually loses in practice.