Noise That Cancels as the Crowd Grows: Privacy for Group-Based Peer-to-Peer Averaging

Peer-to-peer federated learning trains one model across many devices with no central server. It hits a scaling wall because naive designs have every device send its update to every other, so traffic grows with the square of the number of devices. The best current escape, the group-averaging scheme in MAR-FL, gets traffic down to roughly $n \log n$. It does this by having devices average in small groups over several rounds, so that the crowd behind each shared value grows geometrically from stage to stage. That same design creates a privacy problem nobody has measured. In the first stage, a device's update is mixed with only a handful of others. Any curious member of the group can subtract its own contribution from the group average and hand the remainder to a gradient-inversion attack, meaning an attack that reconstructs training images or text from a shared gradient. With groups of two, the remainder is the partner's raw update.

We propose to measure this leak and then close it. Devices add random noise that is protective early, when crowds are small, and that is constructed to cancel exactly once the groups have merged into a large enough crowd. The final averaged model then carries little or no net noise, while every small-crowd message an eavesdropper can see is masked. The claim is that, for group-based averaging, the privacy cost should be paid only where the crowd is too small. This should give reconstruction resistance comparable to per-message differential privacy at utility close to non-private averaging, and it keeps the $n \log n$ traffic.

Why this gap is real

Three bodies of work each cover part of the problem. On communication, MAR-FL reports "up to 10x" less traffic than all-to-all baselines with "same test accuracy" as FedAvg on MNIST over 125 peers. It adds decentralized differential privacy with adaptive clipping, but it never runs an inversion attack against its intermediate group messages. That omission motivates this proposal.

On leakage, the attacks are strong. Deep Leakage from Gradients recovers images by optimizing dummy inputs until their gradients match the shared ones, but it struggles beyond batch size 8. Gradient Inversion with Generative Image Prior searches the latent space of a pretrained image generator instead of raw pixels. It reconstructs from batch 32 and stays effective with "99% of gradient values removed." Concealing Sensitive Samples explains when isolation succeeds: individual gradients must be "low-entanglement," occupying nearly separate directions. That condition is exactly what a small group produces.

On noise design, Mitigating Privacy-Utility Trade-off in Decentralized FL via f-DP supplies our method's closest ancestor. It analyzes correlated noise that cancels across trusted neighbors, and tighter accounting on fixed graphs (ring, torus, hypercube), using f-DP, a way of measuring privacy through how well an attacker can tell two datasets apart. That work assumes a static graph and does not test real inversion attacks. We differ in both respects: our cancellation is tied to a changing schedule of groups, and we evaluate against reconstruction attacks directly.

Core Claim

In a multi-stage group-averaging protocol, the only messages needing protection are those whose crowd, after an observer subtracts what it already knows, is smaller than a target size $m$. The noise can therefore be arranged to cancel at the first stage whose groups reach size $m$.

Concretely, let $z_c$ be the noise vector added by client $c$. Let $G_s(c)$ be the group containing $c$ at stage $s$, where each group at stage $s$ is a union of groups from stage $s-1$. Let $s^*$ be the first stage whose groups have at least $m$ members. Clients draw pairwise-seeded Gaussian noise satisfying

$$\sum_{c' \in G_{s^*}(c)} z_{c'} = 0 \quad\text{and}\quad \mathrm{Var}\Big(\sum_{c' \in G_s(c)} z_{c'}\Big) \ge \sigma_s^2 \ \text{ for all } s < s^* .$$

Here $\sigma_s$ is the noise level needed at stage $s$ to meet a target f-DP guarantee against an observer who removes its own contribution. In words, the noise is zero-sum within each large group but not within any of its smaller sub-groups.

A grid layout makes such noise easy to generate. Take clients that average along one grid axis per stage, and have pairs that first meet at stage $s^*$ share a random seed, adding $+\xi$ and $-\xi$ respectively. An optional small independent noise term remains to give population-level differential privacy on the final model, and it is cheap there because the crowd is all $n$ clients.

Two failure modes need their own handling. The first is churn: a client that drops out leaves its partner's noise uncancelled. We handle this by letting the surviving partner regenerate and subtract the dropped share from the shared seed, borrowing the offline and online split of PICO. The second is collusion. Colluders who hold both seeds of a pair remove that pair's masking. Following the collusion analysis style of Li et al., we will bound the guarantee as a function of the number of colluders $T$.

Research Program

The work has three parts.

The first part is an attack study. We instrument MAR-FL-style group averaging with group sizes $k \in \{2,4,8,16\}$. A curious peer subtracts its own update from each message it receives, and then runs both the pixel-matching attack of Zhu et al. and the generator-based attack of Jeon et al. on the remainder. We record reconstruction quality separately for each stage.

This study also settles a question the literature leaves open about sparsification. Zhu et al. report that pruning more than 20% of gradient values stops pixel matching. Breaking the Communication-Privacy-Accuracy Tradeoff with f-DP proves that ternary sparsification amplifies privacy. Yet Jeon et al. still succeed at 99% sparsity. We will test whether the sparsification that subquadratic systems use to cut bytes actually defends small-group messages against generator-based attacks.

The second part is the mechanism. It includes the schedule-aligned noise construction, f-DP accounting for each stage under $T$ colluders, and dropout recovery.

The third part is the three-way frontier. For each group size and each $m$, we trace bytes per client against reconstruction quality against accuracy. This makes the trade between communication and exposure explicit rather than implicit.

Baselines

The naive reference point is MAR-FL group averaging with no privacy mechanism. This gives the utility ceiling and the leakage floor.

The strongest existing system is MAR-FL with its own decentralized DP and adaptive clipping. We run it at the same group sizes, peer count, data split and number of training rounds. We tune its clipping and noise with the same hyperparameter budget as ours, to the same nominal f-DP level. We beat it if, at matched per-message privacy, we get higher accuracy; or if, at matched accuracy, we get lower reconstruction quality.

The strongest correlated-noise method is the static-graph correlated noise of Li et al., run on a hypercube with matched per-client traffic. Their reported MNIST result is 89.1% test accuracy at $\epsilon=10$, against 86.7% for an RDP-accounted baseline (Li et al.). We must exceed 89.1% on MNIST at $\epsilon=10$ with lower traffic.

We also include ERIS, which splits each update into shards handled by different client-aggregators with error-compensated sparsification. It reduces reconstructions to "unrecognizable noise" with two aggregators. We run it with and without colluding aggregators at equal bytes.

Two ablations isolate the key ingredient. The first adds independent noise of the same per-stage variance without cancellation. The second adds cancelling noise whose pairs are chosen at random rather than aligned to the schedule, so that cancellation may happen too early or too late.

Why not just…?

Why not just run secure aggregation inside each small group? Hiding each member's message only reveals the group sum. But a member knows its own update, so it learns the sum of the other $k-1$ members, and exactly its partner's update when $k=2$. The leak lives in the protocol's outputs, not in the channel. Breaking Secure Aggregation adds that manipulated batch normalization layers recover labels from aggregates at 100% accuracy. We include group-level secure aggregation as a baseline to show that its stage-1 reconstructions remain high quality for small $k$.

Why not just use bigger groups? This is the real threat. Group size is the knob that trades traffic for crowd size, and moderately large $k$ combined with secure aggregation might suffice. The frontier experiment settles it. If bigger groups plus secure aggregation match our leakage at equal bytes and accuracy, the mechanism is unnecessary and we will report that. We expect otherwise, because the generator-based attack of Jeon et al. still works at batch 32, which suggests crowds of that size are not safe.

Why not just apply standard DP-SGD to every message? Noise large enough to stop inversion hurts accuracy. Zhu et al. find that noise stops their attack only above $10^{-2}$, where accuracy degrades. DP Adapters report DP-SGD at about 53% against 96.49% non-private. Our independent-noise ablation is exactly this alternative at matched per-message protection.

Why not just shard updates as ERIS does? Sharding assumes aggregators do not collude. Our collusion sweep over $T$ applies to both methods, and we report where each breaks.

Proposed Experiments

We simulate 125 peers with 20% dropout and 50% participation, mirroring MAR-FL's churn setting. Workloads are MNIST with a CNN, CIFAR-10 with ResNet-9, and 20 Newsgroups with DistilBERT, all with non-i.i.d. splits. Attacks use CIFAR-10, ImageNet 64×64 and FFHQ 32×32, the settings where Jeon et al. report 15.58 dB PSNR against 13.78 dB for pixel matching. We also run label inference.

Metrics are:

  • PSNR and a perceptual distance between reconstructions and the originals, per stage;
  • membership-inference advantage;
  • test accuracy;
  • bytes per client per round;
  • empirical privacy audits that compare the privacy an attacker actually achieves against our f-DP accounting.

Three results decide the question. Stage-1 reconstructions under undefended group averaging must be recognizable at small $k$; if they are not, the premise fails. Our method must push stage-1 PSNR to the level of the independent-noise ablation while its final accuracy stays within a small margin of the no-privacy reference. And at equal bytes, it must dominate the frontier traced by bigger groups with secure aggregation.