In federated learning, many devices jointly train one model by sending the server updates computed on their local data rather than the data itself. Two problems sit on that uplink. The updates are large, so many methods compress them. The updates also leak, because an attacker who sees an update can often reconstruct the training images or sentences behind it. Compression is often presented as a privacy defense as well, and the best evidence says it is not one: a learned attacker trained "through" sign compression still reconstructs CIFAR-10 images at a mean squared error of 0.014 (Learning To Invert). Formal differential privacy, the mathematical guarantee that any one input changes the output distribution only slightly, does protect, but adding noise to every one of millions of coordinates usually ruins accuracy. This proposal turns the compressor itself into the privacy mechanism, and makes it cheap in accuracy by choosing where it operates.
Each client first projects its update onto a random low-dimensional subspace that the server can regenerate from a shared seed. It bounds each projected coordinate and then sends each coordinate as one of three symbols, +1, 0, or −1, drawn at random so that the server's estimate stays unbiased. The randomness of this three-level ("ternary") compressor is the privacy noise. Recent theory shows that such a compressor carries a closed-form privacy guarantee, and that sending more zeros strengthens it (Breaking the Communication-Privacy-Accuracy Tradeoff with f-Differential Privacy). That theory has only been tested on averaging 250-dimensional vectors. The bet here is that random projection is the missing piece that makes it usable for training deep networks. Projection shrinks the dimension in which privacy noise accumulates. It also spreads the update's energy almost evenly across coordinates, so bounding each coordinate discards little signal. And it makes the choice of transmitted coordinates independent of the data, which closes a leakage channel that magnitude-based or loss-aware selection leaves open.
Core Claim
At matched uplink bits and matched per-round privacy budget, seeded-subspace ternary updates will reach higher accuracy than the usual alternative of clipping, adding Gaussian noise, and then compressing. Their guarantee covers attacks that heuristic compression fails against. This includes learned inversion and also active servers that tamper with model weights, because a local differential privacy guarantee holds for whatever model the server sends. Clipping fixes the sensitivity no matter what the weights are.
The mechanism, for client update $\Delta \in \mathbb{R}^D$ in round $t$, is
$$u = \mathrm{clip}_{\infty, c}\!\left(P_t \Delta\right), \qquad \hat{u}_j \in \{-1, 0, +1\},\ \ \Pr[\hat{u}_j = \pm 1] = \tfrac{A \pm u_j}{2B},$$
Here $P_t \in \mathbb{R}^{k \times D}$ is a Gaussian matrix regenerated from a public per-round seed, with $k \ll D$. The constant $c$ bounds each coordinate, and $A \ge c$ and $B \ge A$ set the probability of transmitting a nonzero. The server rescales by $B$, averages over clients, and maps back with $P_t^\top$. This parameterization follows the ternary stochastic compressor of the f-DP paper. Its privacy cost per round depends on $A$, $B$, and $c$, not on $D$. The bit cost is roughly $k \log_2 3$, and less with entropy coding when $B \gg A$. A fresh $P_t$ each round means the subspaces cover the full parameter space over training.
Relation to Prior Work
The f-DP ternary work supplies the method. It derives hypothesis-testing privacy guarantees for discrete compressors and shows that compression can amplify privacy, but only for distributed mean estimation with $N=1{,}000$ users and $d=250$. Its bounds also rely on a central-limit approximation that is loose in small dimensions. The projection dimension $k$ is a knob here, chosen in the thousands, where that approximation is meant to hold.
DP-GRAPE motivates the projection. It shows that under DP noise, cheap Gaussian random projections work as well as SVD subspaces for centralized private training up to OPT-6.7B. It addresses memory, not federated uplinks, and it does not test inversion attacks. ZAMPLING (Trading-off Accuracy and Communication Cost) also trains through a fixed seeded random matrix and reports a 1024-fold communication reduction on MNIST. However, it has no privacy analysis, and its fixed matrix may make inversion easier. This proposal uses a fresh matrix per round and attaches a formal guarantee.
Three papers constrain the design. Learning To Invert rules out any defense evaluated only against optimization attacks, so every configuration here is attacked by an inverter trained through the exact mechanism. Evaluating Gradient Inversion Attacks and Defenses shows that gradient pruning must exceed 99% to stop reconstruction on its own. It also shows that attacks weaken without batch-norm statistics, so the threat model must state which metadata the attacker sees. PEFTLeak shows that parameter-efficient fine-tuning is not private against a weight-tampering server, recovering 85.9% of CIFAR-100 patches at SSIM 0.88. That is the active setting where a formal local guarantee should matter most.
Mixed quantization and discrepancy-aware calibration are the compression-side competitors. The first hides quantization keys and has no formal guarantee. The second picks coordinates by their effect on outputs over a local calibration batch, which may make the transmitted support itself data-revealing.
Research Program
The first part is analysis. Jin et al.'s per-round f-DP guarantee for the ternary compressor will be composed across rounds and client sampling. The analysis will report what this composition actually certifies at the client level against the server. Single-update guarantees will be stated separately, because those are what bound reconstruction from one intercepted update. The analysis will also bound the extra variance from projection and per-coordinate clipping, which governs the accuracy cost of choosing $k$. The expected outcome is an interior optimum in $k$ at fixed bits and budget. Too small a $k$ discards signal through projection. Too large a $k$ spreads a fixed bit and privacy budget over too many coordinates.
The second part is a uniform attack harness. It runs three attackers against every mechanism: optimization-based gradient matching in the setting of Huang et al., the generative-prior attack GIAS (Gradient Inversion with Generative Image Prior), and a Learning To Invert model retrained per mechanism on auxiliary data. A PEFTLeak-style active server on a ViT with adapters is added for the parameter-efficient setting.
The third part tests a secondary question that falls out of the design. Does data-dependent coordinate selection leak through the support alone? An attacker will be given only the indices that Top-$k$ or discrepancy-aware selection chose, with the values withheld.
Baselines
The naive reference point is uncompressed FedAvg without privacy protection. It sets the accuracy ceiling and the worst-case leakage. On CIFAR-10, Huang et al. report 93.37% accuracy undefended with LPIPS 0.45. LPIPS is a perceptual distance, so higher means a worse reconstruction.
The strongest competitor for formal guarantees is clip, add Gaussian noise, then compress with stochastic quantization or Top-$k$. Post-processing keeps its guarantee intact, so it is the obvious way to get privacy and compression together. For the same $(\epsilon, \delta)$ per round, the same uplink bits per round, the same clients and data splits, and the same number of hyperparameter trials (the clipping norm and learning rate tuned by identical grid search), I found no reported result on Lacuna for DP federated training of CIFAR-10 with compression. It will be measured fresh.
The strongest heuristic defense is Huang et al.'s combination of 90% gradient pruning, Intra-InstaHide, and batch size 32. It reports 86.10% CIFAR-10 accuracy with LPIPS 0.73. Beating it means matching or exceeding 86.10% at an LPIPS at least that high under all three attackers, including LTI, which Huang et al. did not evaluate.
On the compression side, discrepancy-aware Top-$k$ reports 29.62% for ViT-tiny on CIFAR-10 at a 0.01 compression ratio, against 21.03% for magnitude Top-$k$. This is the bar for accuracy at extreme compression without privacy. The proposal does not need to beat it outright. It needs to come within a stated margin while adding a guarantee.
Two ablations isolate the key ingredient. The first keeps the same seeded subspace but replaces the ternary compressor with Gaussian noise plus deterministic 2-bit quantization at equal bits and equal epsilon. This is DP-GRAPE's projection ported to federated learning, and it tests whether the ternary mechanism matters beyond the projection. The second applies the ternary compressor directly in the full $D$-dimensional space with random coordinate subsampling. It tests whether projection matters.
Why not just…?
Why not just add Gaussian noise and then compress, since post-processing preserves privacy? This is the real threat. If Gaussian-then-compress matches the proposal at equal bits and epsilon, the contribution shrinks to an accounting convenience. The structural reason to expect a gap is that it pays twice: once in privacy noise and again in quantization error on an already-noisy vector. The ternary mechanism spends a single source of randomness on both. Prior evidence that full-dimensional noise is costly comes from the mixed-quantization paper, where DP baselines fell to 78–85% on MNIST against 96.67% for quantization. Those settings were not matched, though, so the first ablation is the decisive test and will be reported whichever way it comes out.
Why not just use heuristic compression such as sign, Top-$k$, or mixed quantization, which already defeats reconstruction? It defeats only the attacks it was tuned against. Learning To Invert reaches MSE 0.014 under sign compression, where optimization attacks sit near 0.11. The mixed-quantization result was measured only against DLG on LeNet-5. The attack harness retrains LTI through every heuristic and reports whether its advantage survives.
Why not just use large batches, or shard updates across non-colluding aggregators as in ERIS? GIAS keeps reconstruction quality at batch size 32, and PEFTLeak recovers patches at the same batch size. ERIS's guarantee disappears if its aggregators collude. Both are complementary to this proposal, not substitutes for it. ERIS-style sharding will be stacked on top of the ternary updates to check that the gains are additive.
Proposed Experiments
The vision experiments use CIFAR-10 with ResNet-9 and ViT-tiny, and FEMNIST with its natural non-IID writer split. The parameter-efficient setting uses CIFAR-100 with a ViT and adapters. Text uses CoLA, where LTI recovers about 62% of sentences under sign compression. Each experiment sweeps three settings: bits per round (matched to compression ratios of 0.1, 0.01, and 0.001), per-round epsilon (0.5, 1, 2, and 4), and projection dimension $k$.
For each configuration, the outcomes are test accuracy and the reconstruction quality of the strongest of the three attackers, measured by LPIPS, SSIM, MSE, and sentence recovery. These are plotted as accuracy against attack quality at fixed bits.
The first decisive check is that, at equal bits and equal epsilon, the proposal sits strictly above Gaussian-then-compress on the accuracy axis on CIFAR-10 and FEMNIST. If it does not, the claim fails.
The second decisive check is that under PEFTLeak's active weight tampering, patch recovery falls far below 85.9% at a privacy level where accuracy stays within a few points of the non-private adapter baseline. An active server gaining nothing beyond the guarantee would confirm that local guarantees cover tampering in practice and not only on paper.
The third check is the support-only attack. If indices chosen by discrepancy-aware selection let an inverter recover class or attribute information better than chance, while the proposal's seeded random support provably reveals nothing, data-independent selection is shown to be a necessary design rule and not an incidental choice.
The final check tests the analysis. If the empirical accuracy-versus-$k$ curve has the interior optimum predicted by the variance bound, the projection dimension can be set from the bound and not by search.
References
- Learning To Invert: Simple Adaptive Attacks for Gradient Inversion in Federated Learning
- Breaking the Communication-Privacy-Accuracy Tradeoff with $f$-Differential Privacy
- Memory-Efficient Differentially Private Training with Gradient Random Projection
- Trading-off Accuracy and Communication Cost in Federated Learning
- Evaluating Gradient Inversion Attacks and Defenses in Federated Learning
- Gradient Inversion Attacks on Parameter-Efficient Fine-Tuning
- Mixed Quantization Enabled Federated Learning to Tackle Gradient Inversion Attacks
- Enhancing Communication Compression via Discrepancy-aware Calibration for Federated Learning
- Gradient Inversion with Generative Image Prior
- ERIS: Enhancing Privacy and Communication Efficiency in Serverless Federated Learning