Federated learning trains one model across many devices that each send gradient updates (directions in which to nudge the weights) to a server. To save bandwidth, practical systems compress those updates aggressively, for example by sending only the largest 1% of coordinates. Biased compressors like this only converge when paired with error feedback: each device keeps a private memory of everything it has not yet sent and adds it to the next update. This proposal argues that this memory breaks machine unlearning, the removal of a specific user's data from a trained model. When a device is asked to forget some of its samples, its error-feedback memory still holds gradient mass computed on those samples. That mass keeps flowing into the global model for many rounds after the unlearning step "finishes." The same mechanism has a second effect: error feedback is built so that the running sum of compressed messages tracks the running sum of true gradients. Any privacy that compression seems to give the unlearning update is therefore temporary, and an eavesdropper who adds up a few rounds sees almost the uncompressed signal.
We propose to measure both effects and to test a small, buffer-aware unlearning protocol against much simpler fixes. The success criterion is the standard one for unlearning: a membership inference attack, which guesses whether a sample was in the training set, should be no better than chance on forgotten samples. We also require that forgotten images cannot be reconstructed from the client's transmissions. The claim to a colleague, in two sentences: compressed federated training leaves a hidden copy of to-be-forgotten gradients in each device's error-feedback memory, which can quietly undo unlearning and can turn compressed messages back into reconstructable ones. We quantify this and determine whether a targeted surgery on that memory is needed, or whether simply discarding it suffices.
Core Claim
The relevant state is the error-feedback recursion analyzed in Linearly Converging Error Compensated SGD:
$$e^{k+1} = e^k + \gamma g^k - v^k, \qquad v^k = \mathcal{C}(e^k + \gamma g^k),$$
where $g^k$ is the client's gradient at round $k$, $\gamma$ the step size, $\mathcal{C}$ the compressor (say Top-$k$), $v^k$ the message actually sent, and $e^k$ the unsent residual. Summing over rounds gives $\sum_k v^k = \gamma \sum_k g^k - e^{K}$. The server's cumulative view therefore differs from the uncompressed history only by the current buffer. This identity makes error feedback converge. It also has two consequences for unlearning that, as far as the research file shows, nobody has examined. The first we call resurrection: after a client unlearns samples $S_f$, the residual $e^K$ still contains components of $\nabla \ell(S_f)$ from earlier training rounds, and these drain into the model later. If they matter, membership inference AUC on $S_f$ should rebound over the rounds after unlearning. We expect the effect to grow with compression strength, because a sparser $\mathcal{C}$ leaves a larger buffer. The second we call transient obfuscation: the forgetting client's compressed unlearning messages, summed over the few rounds it takes to drain them, approximate the uncompressed update. Attacks that already work on uncompressed unlearning updates should then recover most of their fidelity.
Positioning Against Prior Work
The motivating evidence comes from attacks on unlearning updates. DRAUN reconstructs forgotten images from federated unlearning updates using a surrogate unlearning loss and disjoint initialization of dummy inputs. Verifiably Forgotten? inverts gradient differences sent as proof of unlearning. Learn What You Want to Unlearn inverts before/after parameter differences in centralized model-as-a-service settings. All three assume full-precision updates. None of them asks what a compressed, error-compensated pipeline does to the signal, which is the regime most bandwidth-limited deployments actually run.
Learning To Invert supplies the attack methodology we reuse. It trains a network on auxiliary data to invert gradients and treats the compressor as part of the data pipeline, which beats sign compression and Gaussian noise in ordinary federated learning. It does not study unlearning updates or error-feedback loops. We extend it by putting the full error-feedback recursion in the attacker's training loop and by feeding it multi-round sums.
On the unlearning side, Subspace based Federated Unlearning (SFU) is the federated method we build on. The forgetting client runs gradient ascent, and the server projects that update away from the dominant gradient directions of the remaining clients. SFU sends full-precision updates and never models compression state. Learn to Unlearn for Deep Neural Networks is its centralized counterpart. Towards Unbounded Machine Unlearning constrains how far we may push forgetting: it shows that over-forgetting is itself detectable by membership inference, so our target is the forget-set error of a retrained model, not maximal error.
BlindU and CRFU compress internal representations through an information bottleneck to resist reconstruction. They compress activations, not transmitted gradients, so they are orthogonal to the buffer question.
On compression, vqSGD gives unbiased randomized vector quantizers that need no error feedback and carry intrinsic differential privacy. Distributed Mean Estimation with Limited Communication gives the random rotation trick for unbiased quantization. These are the natural "remove the buffer entirely" alternatives.
We also checked one tension the file flags: lossy compression supposedly spills energy back into the protected retain subspace. We think that problem mostly dissolves. If the server applies the projection after decompressing, linearity gives $\mathbb{E}[P\,\mathcal{C}(g)] = P g$ for any unbiased compressor $\mathcal{C}$ and projection $P$. Projecting can only shrink variance. We therefore do not build on that gap and treat server-side projection as standard.
Research Program
The work has three parts. The first is a measurement study of resurrection. We train with compressed, error-compensated federated SGD and issue sample-level unlearning requests: the client stays in the federation but must forget part of its data. We then track forget-set membership inference AUC, forget-set accuracy, and the norm of the forget-attributable component of each buffer for 50 rounds afterward. To measure attribution, we project the buffer onto the span of per-sample forget gradients, which is cheap because forget sets are small. For client-level unlearning, the departing client's buffer disappears with it. However, systems that also compress the server-to-client downlink keep a server-side buffer that mixes in the departed client's contributions. We measure that path separately.
The second part is an adaptive attack study of transient obfuscation. We run DRAUN's surrogate-loss inversion on single compressed messages and on sums over $m$ consecutive unlearning-round messages for increasing $m$. We also train a Learning-To-Invert style inverter with the exact compressor-plus-buffer simulator in the loop.
The third part is a buffer-aware protocol with three ingredients. Buffer surgery removes the component of the client's buffer lying in the span of current per-sample forget gradients:
$$e \leftarrow e - Q_f Q_f^\top e,$$
where $Q_f$ is an orthonormal basis of those gradients. The rest of the buffer, which carries retain progress, is kept. Second, the forgetting update is sent once with a randomized unbiased vector quantizer and no error feedback, as in vqSGD, so that no unsent residual of the ascent step lingers to be drained or summed. Third, the server applies SFU-style projection after decompression and stops at the retrained model's forget-set error, following the stopping logic of Towards Unbounded Machine Unlearning.
Baselines
The naive reference point is compressed error-feedback training followed by SFU unlearning with no change to buffers or messages. Comparing it with uncompressed SFU on identical splits isolates what compression state alone does to unlearning.
The gold standard is retraining from scratch without the forget samples, using the same compressor and the same number of rounds. Its membership inference AUC and forget error define the target, not 0.50 per se.
The strongest existing federated method is SFU run with full precision. SFU reports only about a 1 percentage point clean-accuracy drop on MNIST, against roughly 47 points for unconstrained ascent (Subspace based Federated Unlearning). We found no reported membership inference result for SFU on Lacuna and no reported result on Lacuna for any federated unlearning method under gradient compression, so we will produce both. The nearest centralized privacy figure is projected-gradient unlearning's membership inference AUC of 0.5340 on CIFAR-10 (Learn to Unlearn for Deep Neural Networks). For reconstruction, the reference attack fidelity is DRAUN's SSIM of 0.6407 and PSNR of 21.11 on CIFAR-10 under ascent-based unlearning (DRAUN). Learning To Invert reports MSE 0.014 under sign compression where optimization attacks sit near 0.11 (Learning To Invert).
Every method gets the same client splits, the same model, the same unlearning-round budget, and the same hyperparameter search budget, with stopping tuned on a held-out validation set as in SCRUB+R. "Beating" the strongest competitor means three things at matched bits transmitted. The protocol must stay within the retrained model's forget-set membership inference AUC over all 50 post-unlearning rounds, where naive compressed SFU is predicted to drift. It must reduce DRAUN and adaptive-inverter SSIM on multi-round sums well below the uncompressed-attack level. And it must stay within one point of full-precision SFU's test accuracy.
Ablations remove each ingredient in turn: buffer surgery replaced by full buffer reset, the unbiased one-shot message replaced by error-feedback Top-$k$, and server-side projection removed.
Why not just…?
Why not just zero the buffer when an unlearning request arrives? This is the most serious threat, and it is a planned ablation, not a strawman. Zeroing certainly removes resurrection. Its cost is the retain progress stored in the buffer, which at 1% Top-$k$ can equal many rounds of gradient. That may slow convergence or cause an accuracy dip. Whether this matters is an empirical question we cannot answer from the file. If full reset matches surgery on accuracy recovery and membership inference across compression ratios, we will report that the simple fix suffices and that the contribution is identifying the failure mode. Resetting does nothing about the server-side downlink buffer under client-level unlearning, which needs separate handling.
Why not just send the unlearning update in full precision once? Unlearning requests are rare, so the bandwidth cost of one uncompressed round is small. We concede that the communication argument alone is weak here. However, full precision hands an honest-but-curious server exactly what DRAUN inverts at SSIM 0.6407. It also leaves the training-time buffers, the source of resurrection, untouched. Our experiments include full-precision-plus-reset as a baseline. If it matches our protocol on both membership inference and reconstruction, it is the recommended practice.
Why not just add Gaussian differential privacy noise to the unlearning messages? The file's evidence is consistently negative. DRAUN reports a massive accuracy drop from noise defenses. Learn What You Want to Unlearn reports 5–15% validation accuracy loss for meaningful privacy. Learning To Invert reconstructs through $\sigma=0.1$ noise at MSE 0.012. We include noise calibrated to match our protocol's accuracy so the comparison is at equal utility.
Why not just drop error feedback and use unbiased compressors everywhere? Unbiased quantizers such as vqSGD's cross-polytope pay variance of order $d$ for $O(\log d)$ bits. That likely slows training enough that practitioners keep biased compressors with error feedback. We run vqSGD-only training as a baseline and report rounds to target accuracy. If it trains competitively on our benchmarks, the buffer problem has an easy architectural escape, and we will say so.
Proposed Experiments
We use FEMNIST from LEAF with natural per-writer splits, plus CIFAR-10 and CIFAR-100 with 100 clients under IID and Dirichlet non-IID splits, matching DRAUN and SFU. Models are a small ConvNet and ResNet-18. Compressors are Top-$k$ with error feedback at 10%, 1% and 0.1%, sign compression with error feedback, rotated stochastic quantization, and vqSGD cross-polytope and Hadamard codebooks. Unlearning settings are sample-level, including high-memorization samples (the regime where Sharpness-Aware Machine Unlearning finds attacks bite hardest), and client-level with and without downlink compression. Metrics are test and retain accuracy, forget error relative to retraining, LiRA-style membership inference AUC tracked per round after unlearning, reconstruction SSIM, PSNR and LPIPS under DRAUN and the adaptive inverter, and total bits transmitted.
Three outcomes are decisive. First, resurrection is real if naive compressed SFU's forget-set membership inference AUC rises above the retrained model's band in later rounds, rising more steeply as compression gets sparser, while uncompressed SFU stays flat. A null result here would refute half the thesis, and we would publish it as such. Second, transient obfuscation is real if DRAUN's SSIM on summed error-feedback messages approaches its uncompressed value within a handful of rounds, while single messages look safe. Third, buffer surgery earns its place only if it beats full reset on post-unlearning accuracy recovery at equal membership inference AUC. If it does not, we recommend the reset and drop the surgery.
References
- Linearly Converging Error Compensated SGD
- DRAUN: An Optimization-Agnostic Data Reconstruction Attack on Federated Unlearning
- Verifiably Forgotten? Gradient Differences Still Enable Data Reconstruction in Federated Unlearning
- Learn What You Want to Unlearn: Unlearning Inversion Attacks against Machine Unlearning
- Learning To Invert: Simple Adaptive Attacks for Gradient Inversion in Federated Learning
- Subspace based Federated Unlearning
- Learn to Unlearn for Deep Neural Networks: Minimizing Unlearning Interference with Gradient Projection
- Towards Unbounded Machine Unlearning
- BlindU: Blind Machine Unlearning without Revealing Erasing Data
- CRFU: Compressive Representation Forgetting Against Privacy Leakage on Machine Unlearning
- vqSGD: Vector Quantized Stochastic Gradient Descent
- Distributed Mean Estimation with Limited Communication
- LEAF: A Benchmark for Federated Settings
- Sharpness-Aware Machine Unlearning