What Zero-Sum Masks Hide Over Many Rounds: Mask-Invariant Gradient Inversion in Decentralized Optimization

Several privacy schemes for decentralized optimization let each agent add a secret "mask" to its private cost function. The masks are chosen so that they sum to zero across the network, so the global problem is unchanged and the algorithm still converges to the exact answer. These schemes are judged secure because a single intercepted gradient looks like noise. Decentralized optimization means a network of agents jointly minimizing a sum of private costs by exchanging messages only with graph neighbors. Our point is simple. The mask is fixed for the whole run, but an eavesdropper sees hundreds of rounds. A fixed linear mask adds the same constant to every gradient, so subtracting two consecutive gradients removes it exactly. In gradient tracking, a widely used decentralized algorithm, the transmitted messages are already built from such differences, so the mask disappears from the protocol after round zero.

The proposal turns this observation into a measured result and a repaired defense. We build a gradient inversion attack that never estimates the mask. Gradient inversion means reconstructing training data by searching for inputs whose gradients match the observed ones. Our attack matches gradient differences across rounds instead of raw gradients. We then characterize exactly what such masking still protects: the affine part of each local cost and nothing more. We test this on the same benchmarks that zero-sum schemes used to claim security. Finally, we study per-round refreshed zero-sum masks as a fix, against the strongest simpler alternative, which is decaying independent noise.

Core Claim

Take the Function Sharing protocol of Preserving Statistical Privacy in Distributed Optimization. Each agent $i$ replaces its cost $f_i$ with $\tilde f_i(x) = f_i(x) + \langle u_i, x\rangle$, where the vectors $u_i$ are random and sum to zero across agents. Any observer who recovers masked gradients at two iterates $x_t$ and $x_{t+1}$ obtains

$$\nabla \tilde f_i(x_{t+1}) - \nabla \tilde f_i(x_t) = \nabla f_i(x_{t+1}) - \nabla f_i(x_t),$$

which is the unmasked gradient difference. In gradient tracking, agent $i$ sends neighbors a tracker $v_i^{t+1} = \sum_j W_{ij} v_j^t + \nabla \tilde f_i(x_i^{t+1}) - \nabla \tilde f_i(x_i^t)$, where $W$ is the mixing matrix of the graph. The mask therefore enters only through the initialization $v_i^0$.

Our claim has two halves. The first is a negative result. Linear zero-sum masking hides exactly the affine component of $f_i$. That matches the "affine privacy" notion of the original Function Sharing paper, read literally. For machine-learning losses, however, the data enter through curvature, and curvature survives differencing. Concretely, on a quadratic cost $x^2 + ax$ the private coefficient $a$ stays hidden. On logistic regression or a small convolutional network, a stream of gradient differences should suffice to reconstruct the training images.

The second half concerns nonlinear masks. These include the structured perturbations of EFPSN in Private and Accurate Decentralized Optimization via Encrypted and Structured Functional Perturbation, or any static mask drawn from a known family with $p$ parameters. Here the same logic generalizes. An attacker observing more than $p$ rounds can fit the mask parameters jointly with the data. Leakage is then governed by whether the data are identifiable modulo that mask family. Masking alone does not set it.

Relation to Prior Work

Two zero-sum schemes motivate this work and are its main targets. Function Sharing builds masks by exchanging neighbor noise and proves statistical privacy while honest nodes stay connected. EFPSN generates the masks under Paillier encryption, proves $(\epsilon,\delta)$-differential privacy, and shows that single-round iDLG reconstructions on MNIST blur as mask magnitude grows. Neither paper tests an attacker who uses the time series of messages. That is the attacker we build.

Two attack papers supply the reasoning that multi-round observation breaks static protection. Temporal Gradient Inversion Attacks With Robust Optimization fuses many gradient snapshots and recovers batches up to 128 on ResNet-18. Gradient Disaggregation un-sums secure aggregates across rounds, raising CIFAR-10 reconstruction from 6.1 dB to 36.5 dB PSNR (peak signal-to-noise ratio). Both work against server-side federated learning. Neither handles masks that are designed to cancel, nor decentralized trackers.

Peer-to-peer leakage itself is established by two papers. On the (In)security of Peer-to-Peer Decentralized Machine Learning shows that neighbors in plain decentralized averaging can isolate a victim's contribution. It does not study gradient tracking or functional masking. Topology-Dependent Privacy Bound For Decentralized Federated Learning proves that a passive adversary learns at most the sum of private data within each "honest component," a connected set of honest nodes. We reuse this result to set the threat model for the refreshed-mask defense.

Gradient Inversion Attacks Beyond SGD supplies the method template. It matches optimizer updates rather than raw gradients, and we match differences instead.

Research Program

The threat model is a passive eavesdropper, or a set of colluding neighbors. It observes agent $i$'s outgoing messages and those of $i$'s neighbors, which is what is needed to strip off the mixing term and isolate $i$'s local contribution. We separately evaluate the weaker case where the attacker covers only part of the neighborhood. In that case Li et al.'s component-sum bound should apply.

The attack has three parts. First, it recovers the local gradient-difference stream from intercepted messages under three protocols: decentralized gradient descent with Function Sharing, EFPSN, and gradient tracking with linear masks. Second, for full-batch local data, it runs inversion by optimizing dummy inputs and labels to minimize a cosine or Euclidean distance between dummy and observed differences, summed over a window of $T$ rounds. For minibatches, it jointly optimizes the two batches that enter each difference. Third, for nonlinear static masks, it adds the mask parameters as extra unknowns fitted over the window.

On the analysis side, we characterize identifiability in two settings. For linear and logistic models we derive which directions of the data are pinned down by the span of observed gradient differences. We expect that span to have full rank, in the data's coordinates, after a number of rounds on the order of the feature dimension. For quadratic costs we prove that the affine coefficient is exactly the protected part.

The repair is per-round refreshed zero-sum masks. Neighbors derive fresh masks each round from shared pseudorandom seeds agreed once at setup, so the per-round cost is computation only, with no extra messages. Because each round's masks sum to zero, the network-average tracker is preserved and convergence stays exact if the mask magnitude decays on a summable schedule. A neighbor $j$ knows its own shares with $i$, so protection against $j$ comes only from $i$'s shares with other neighbors. That ties the defense's guarantee directly to honest-component size. We will analyze the tradeoff between decay rate and residual consensus error, the error that EFPSN's exact-consensus assumption ignores.

Baselines

The naive reference point is the unprotected protocol: plain decentralized gradient descent and plain gradient tracking under the same inversion attack. It sets the ceiling on reconstruction quality. Concealing Sensitive Samples against Gradient Leakage in Federated Learning reports 59.20 dB PSNR on MNIST without defense in its federated setting, which gives a scale.

The strongest existing defense on the accuracy side is EFPSN. It reports that logistic-regression accuracy on MNIST stays high up to noise magnitude $\gamma = 10^4$, whereas non-zero-sum perturbation collapses around $\gamma = 10^0$. It reports no PSNR for its iDLG experiments: there is no reported result on Lacuna for EFPSN reconstruction PSNR. We will therefore re-run EFPSN at exactly its MNIST logistic-regression and LeNet settings and mask magnitudes. Against it we will run the iDLG attack it was tested with and our difference-matching attack, using the same model, iterate sequence, and number of attacker optimization steps. "Beating" this baseline means one specific result. At a $\gamma$ where EFPSN's single-round iDLG output is unrecognizable, our attack should reach reconstruction quality within a few dB of the unprotected ceiling, with SSIM (a perceptual image-similarity score) above 0.5.

The strongest competing defense is locally differentially private gradient tracking from Locally Differentially Private Gradient Tracking for Distributed Online Learning over Directed Graphs. It adds independent noise with decaying variance and still converges exactly. It reports accuracy and speed gains only qualitatively: no reported absolute number found on Lacuna. We also include DP-RECAL from Differentially Private Decentralized Optimization With Relay Communication, which throttles how often each node transmits. It likewise gives no absolute number in the file. Each defense gets the same tuning budget over its noise or decay schedule. We then compare attack PSNR and SSIM at matched final test accuracy on MNIST, Fashion-MNIST, and CIFAR-10.

Two ablations isolate our key ingredients. Attack-side, the same attacker matches raw masked gradients instead of differences, everything else fixed. Defense-side, refreshed masks are compared with static masks of equal magnitude.

Why not just…?

Why not just use independent decaying noise, as LDP gradient tracking already does? This is the real threat to the defense half of the proposal. It already achieves exact convergence with a finite privacy budget. The case for zero-sum refreshing is that noise which cancels across the network does not bias the average. That should permit far larger per-round perturbation at equal accuracy, which is the EFPSN gap of $10^4$ versus $10^0$, though that gap was measured only for static masks. The temporal-attack paper also reports that its robust multi-snapshot fusion filters zero-mean noise. If the matched-accuracy comparison shows that decaying independent noise resists the difference attack as well as refreshed masks do, we will report that and drop the defense claim. The attack and identifiability results stand regardless.

Why not just run secure aggregation among neighbors? Gradient Disaggregation shows that multi-round aggregates can be un-summed with 100% recovery of who participated when, given side information. Pasquini et al. report heavy overhead for secure aggregation in peer-to-peer graphs. More directly, secure aggregation over a neighborhood leaks the same component sums that Li et al. bound. Refreshed masks aim at that same guarantee at lower cost, and we will check whether they reach it rather than assume it.

Why not just refresh the static mask every round, without any new attack? Because the field has no evidence that static masks fail. Their published evaluations use single-round attacks only. Without an attack that demonstrably breaks the static version, a refresh is an unmotivated cost. The raw-versus-difference ablation is exactly what establishes the need.

Decisive Checks

Three outcomes would settle the core claim, and the first is falsifiable on the quadratic case. On synthetic quadratic networks from the Function Sharing setting (3 agents up to 1,024 nodes), the difference attack should recover curvature exactly and the linear coefficient at chance. If it recovers the coefficient, the identifiability analysis is wrong.

On MNIST logistic regression and LeNet under EFPSN, reconstruction quality from difference matching should be essentially flat in $\gamma$. Single-round iDLG quality should fall with $\gamma$, as EFPSN reports. Under linearly masked gradient tracking, reconstruction from rounds $t \ge 1$ should match the unmasked tracker.

For the refreshed-mask defense, attack PSNR should track honest-component size $m$, falling as $m$ grows in the way Li et al. report for their passive bound. It should do so at final accuracy equal to the unmasked protocol, and the head-to-head comparison against LDP gradient tracking at matched accuracy decides whether the defense is worth keeping.