In decentralized learning, devices train a shared model by exchanging updates with neighbors in a communication graph. They never share raw data. Correlated noise cancellation protects these updates: pairs of neighbors agree on a secret random vector, one adds it and the other subtracts it, so each message looks noisy while the noise sums to zero across the network. The Privacy Power of Correlated Noise in Decentralized Learning (DECOR) proves this approach matches the accuracy of a trusted central server under formal differential privacy on any connected graph. A curious neighbor, however, can subtract the noise it shares with you. What remains is your update plus noise from your other links. That noise is spread evenly over millions of parameter directions. When devices hold very different data, which is the normal case, your update is not spread evenly. It concentrates in a few directions that reflect your own classes and writing style. An attacker who projects your message onto those few directions throws away most of the noise and keeps most of your signal.
We propose to test this concern, then fix it. First, we will measure whether gradient inversion (reconstructing training images from updates) by a curious neighbor becomes easier as data heterogeneity grows, holding the formal privacy budget fixed. Second, we will reshape the canceling noise so its energy sits in the directions where a device's update departs from its neighborhood. The directions are estimated only from messages that device has already released, so the formal guarantee is untouched. Because the noise cancels in the network average, putting it where the attacker looks should cost little accuracy. The question is whether it buys more protection than simply adding more evenly spread canceling noise.
Core Claim
Our claim has two parts. The first is a measurement. At a fixed formal privacy level under DECOR's secret-based local differential privacy, image reconstruction by a curious neighbor improves as label skew increases. The cause is that a skewed device's deviation from the network mean occupies a subspace of dimension $k$ much smaller than the parameter count $d$. Isotropic noise of total energy $\sigma^2 d$ puts only $\sigma^2 k$ of that energy in any fixed $k$-dimensional subspace. Projection therefore raises the signal-to-noise ratio by roughly $d/k$ in the directions that carry the device's private information.
The second part is a mechanism. Each device $i$ keeps a basis $U_i$ ($d \times k$) of the top principal directions of its recent released messages minus its received neighbor messages. On each link $(i,j)$, the canceling vector becomes
$$v_{ij} = \alpha\, U_i \xi_{ij} + \beta\, g_{ij},$$
where $\xi_{ij}$ and $g_{ij}$ are Gaussian vectors drawn from the link's shared seed. $\alpha$ and $\beta$ split a fixed canceling-noise budget between the shaped part and the isotropic part. DECOR's original noise is the case $\alpha = 0$.
The formal guarantee survives for two reasons. $U_i$ is computed from already-privatized outputs, so it is post-processing. The added component is independent noise. Neighbor $j$ also sees $i$'s released messages, so both endpoints can compute $U_i$ and cancellation stays exact. The shaped noise on link $(i,j)$ does not protect $i$ from $j$, because $j$ knows $v_{ij}$. Protection from $j$ comes from $i$'s other links. This is why DECOR's requirement that the graph remain connected after removing any one node still applies.
Research Program
The work builds on three lines of prior work, each playing a different role.
The correlated-noise papers supply the base mechanism and motivate the gap. DECOR and Low-Cost Privacy-Preserving Decentralized Learning (Zip-DL, which makes each device's outgoing noise sum to zero in a single round) evaluate privacy only through formal accounting or membership inference. Membership inference asks whether a record was used in training, not whether it can be reconstructed. Neither varies data heterogeneity as a factor in leakage. Unified Privacy Guarantees for Decentralized Learning via Matrix Factorization (MAFALDA-SGD) optimizes noise correlations across a graph and across time, and tests on the naturally heterogeneous FEMNIST benchmark. It shapes noise for accounting-optimal utility, not for data-dependent attack directions, and its offline optimization scales as $O(n^3T + n^2T^3)$. Our shaping is online and costs a rank-$k$ decomposition per device per refresh. DMM (DMM: Distributed Matrix Mechanism) addresses server-based federated learning with a single global model. It is related but outside our peer-to-peer threat model.
The inversion literature supplies the attack and constrains the design. We reuse the cosine-distance inversion with total-variation smoothing from Inverting Gradients, which recovers images from trained networks and whose reconstructions drift toward class prototypes. That drift is the bias we expect label skew to amplify. SoK: Gradient Leakage in Federated Learning constrains us in three ways. It finds that multiple local steps push reconstructions above LPIPS 0.1 (a perceptual distance under which images stay recognizable), that inversion fails above small resolutions, and that 2-bit quantization defeats current attacks. We therefore work at 32×32 resolution with one local step per round, which is DECOR's regime. There, differencing a neighbor's consecutive models gives an approximate gradient. We treat multi-step training and quantization as baselines rather than ignore them.
The heterogeneity literature supplies the data regimes. We use the 2-classes-per-client CIFAR-10 split from Exploiting Shared Representations for Personalized Federated Learning (FedRep), natural writer skew in FEMNIST from LEAF, and feature shift in the Digits benchmark from FedBN. FedRep lists the privacy of shared representations as future work. We include a decentralized FedRep variant, with correlated noise on the shared body only, as a secondary setting.
Implementation extends the official DECOR code (DECOR repository) inside decentralizepy, on ring, torus, and Erdős–Rényi graphs.
There are two attackers. The generic attacker is a curious neighbor. It subtracts its shared noise and runs cosine inversion on the residual difference of consecutive models. The subspace attacker additionally projects the residual onto a subspace estimated from public model gradients of each candidate class before inverting. We also run an adaptive attacker who knows $U_i$, since $U_i$ is computable from public messages. This attacker inverts using only the component orthogonal to $U_i$. The shaping idea stands or falls on this attacker: it works only if little recoverable signal lives outside $U_i$.
Baselines
The naive reference is decentralized SGD with no noise. It sets the attack ceiling and the accuracy ceiling. Without it we cannot tell whether reconstruction fails because of the defense or because the attack fails anyway under gossip averaging. Next is independent local noise at the same formal budget. DECOR reports this losing nearly 10 accuracy points on MNIST at low budgets while DECOR stays within 1% of centralized accuracy. It anchors the utility cost of not correlating.
The strongest existing method is DECOR itself, run through its released code with identical graphs, data splits, learning-rate grids, and seeds. We report its privacy budget under its own accounting. For its empirical resistance to reconstruction there is no reported result on Lacuna, so our DECOR measurements are new. Zip-DL is a second strong competitor. It reports up to 59% higher CIFAR-10 accuracy than Muffliato at equal membership-inference protection. No reported reconstruction result was found on Lacuna for it either. We will run it at matched privacy against the same attackers.
The ablation that isolates our key ingredient is isotropic-boosted DECOR. It uses the same total extra canceling-noise energy as the shaped variant, spread evenly, so that test accuracy matches within 0.5 points. A second ablation uses a random rank-$k$ subspace in place of $U_i$, which separates "direction matters" from "concentration matters." An oracle variant builds $U_i$ from the device's raw private gradients. It is not formally private, and it serves only as an upper bound on what shaping can achieve.
All methods get the same tuning budget, a fixed grid over learning rate and noise split, chosen on validation accuracy alone. None is tuned against the attack.
Beating DECOR means one of two things at matched test accuracy and matched formal budget. Either median neighbor-reconstruction LPIPS rises past 0.1 where DECOR's stays below it, or PSNR drops by a margin larger than the seed-to-seed spread, under the adaptive attacker.
Why not just…?
Why not just turn up the ordinary canceling noise? Since the noise cancels in the average, more of it is nearly free at the network level. This is the most serious threat to the proposal. On sparse graphs, DECOR notes that leftover noise remains before mixing completes, so extra variance does cost accuracy there. The isotropic-boosted ablation exists to settle this question. If it matches the shaped variant's reconstruction resistance at equal accuracy, we drop shaping and report the heterogeneity-leakage measurement as the contribution.
Why not just quantize updates to 2 bits? The SoK reports that this defeats current inversion attacks with under a 5% accuracy drop. It carries no formal guarantee, and Breaking Secure Aggregation shows compression does not stop label recovery. We run quantized decentralized SGD as a baseline and add the subspace attacker. If quantization alone holds LPIPS above 0.1 at under 5% accuracy loss on our skewed splits, we say so plainly. The case for our method then rests on its formal guarantee.
Why not just take several local steps? The SoK finds that multi-step updates already blur reconstructions. Inverting Gradients reports attacks surviving federated averaging, and FedRep shows that extra local steps worsen drift under label skew. We run 1-, 5-, and 20-step variants and report accuracy and attack quality together.
Why not just keep the classifier head local, as FedRep does? This removes the most class-revealing parameters from messages. However, Inverting Gradients shows fully connected layer inputs are analytically recoverable, and the shared body remains unprotected by any formal guarantee. Our secondary setting measures exactly how much head-locality lowers reconstruction on its own, with and without correlated noise.
Proposed Experiments
The first experiment is a heterogeneity sweep. On CIFAR-10 we vary label skew from IID to 2 classes per device. We also use FEMNIST's natural writer split and the Digits feature-shift benchmark. Graphs are rings and random graphs of degree 2 to 8, at two privacy budgets. For each setting we record test accuracy, reconstruction PSNR and LPIPS for every attacker against DECOR, and the effective dimension $k$ of each device's deviation subspace. The decisive check is directional. Under DECOR, reconstruction quality should increase with skew at fixed budget, and the gain should track $d/k$. If reconstruction does not improve with skew, or no attacker beats LPIPS 0.1 even without noise, the motivating hypothesis fails and we report that as a negative result.
The second experiment tests shaping. We compare shaped DECOR against isotropic-boosted DECOR, the random-subspace ablation, and the oracle at matched accuracy across refresh intervals and ranks $k$. The decisive check is the adaptive attacker. Shaping must beat isotropic boosting when the attacker knows $U_i$ and uses only the orthogonal component. If the orthogonal component alone supports good reconstructions, shaping is only security by obscurity, and we will say so.
The third experiment handles weak points of the threat model. We rerun both experiments under 10–25% device dropout, the range Zip-DL tolerates, and with two colluding neighbors. Both conditions are known to weaken cancellation, and we want to know whether shaping degrades gracefully or collapses back to DECOR's behavior.
References
- The Privacy Power of Correlated Noise in Decentralized Learning
- Low-Cost Privacy-Preserving Decentralized Learning
- Unified Privacy Guarantees for Decentralized Learning via Matrix Factorization
- DMM: Distributed Matrix Mechanism for Differentially-Private Federated Learning Based on Constant-Overhead Linear Secret Resharing
- Inverting Gradients - How easy is it to break privacy in federated learning?
- SoK: Gradient Leakage in Federated Learning
- Exploiting Shared Representations for Personalized Federated Learning
- LEAF: A Benchmark for Federated Settings
- FedBN: Federated Learning on Non-IID Features via Local Batch Normalization
- Breaking Secure Aggregation: Label Leakage from Aggregated Gradients in Federated Learning