Private Memory for Federated Learning That Does Not Wait for Slow Devices

In federated learning, many phones or hospitals train one model together by sending the server model updates instead of raw data. When some devices are slow or offline, a server that waits for everyone stalls. A server that drops the slow devices biases the model toward whoever answers fastest. The best-analyzed fix is to be impatient but remember. Fast Federated Learning in the Presence of Arbitrary Device Unavailability (MIFA) keeps a table holding the most recent update from every one of the $N$ devices. Each round it averages fresh updates with the remembered ones, and this achieves minimax-optimal convergence with no assumptions about when devices show up. That table, however, is a list of individual, unaggregated client updates that sits on the server for many rounds. Direction-matching attacks already reconstruct recognizable images from averaged updates (Inverting Gradients), so a single stored client update is close to the best case for an attacker.

This note proposes moving the memory off the server. The server's update only ever needs the sum of the remembered updates, not the individual entries. Each device can therefore keep its own last update locally and, when it returns, send only the change: its new update minus its old one. These changes are combined under secure aggregation, a cryptographic protocol that lets the server learn only the sum of what several clients send. We add one rule on top: the server does not unmask a sum until at least $k$ devices have contributed. The result is algebraically identical to MIFA, so it inherits MIFA's convergence guarantee exactly. It also removes MIFA's $O(Nd)$ server storage for a model with $d$ parameters. The scientific question is how large $k$ must be before inversion attacks fail on these sums, and what that minimum group size costs in wall-clock time when the stragglers arrive alone.

Core Claim

Write $G_i$ for client $i$'s most recent local update. MIFA's server step is

$$w_{t+1} = w_t - \frac{\eta}{N} S_t, \qquad S_t = \sum_{i=1}^{N} G_i,$$

where $\eta$ is the learning rate. When a set $A_t$ of clients reports in round $t$, the running sum changes by

$$S_t = S_{t-1} + \sum_{i \in A_t} \big(G_i^{\text{new}} - G_i^{\text{old}}\big).$$

Only this window sum of differences needs to reach the server. We make three claims.

First, MIFA's per-client table is a severe and measurable privacy exposure. Standard inversion attacks applied to its entries should come close to the no-defense reconstruction quality reported for single clients.

Second, simply having clients send their differences in the clear does not help. The server can add up a client's successive differences over time, the sum collapses to that client's latest update, and the full update is recovered. Secure aggregation over the window is therefore essential rather than decorative.

Third, a window sum of $k$ differences is a mixture of $2k$ local batches with opposite signs, computed against two or more different past model versions. This should be much harder to invert than a same-size synchronous batch. If so, a small $k$ restores protection against an honest-but-curious server. The alternative is that attackers who know every past model version, which the server does, can unmix it anyway. The existing large-batch attacks were built for a single model version, so neither outcome is settled.

The gradient-inversion literature enters here as the measuring instrument that sets $k$. The unavailability problem is the regime, and the central server holding memory is the vulnerability being removed.

Relation to Prior Work

Three lines of work motivate the gap. MIFA supplies the algorithm we reproduce exactly. It never evaluates leakage, and it lists server storage across millions of clients as a limitation, which client-held memory removes. Beyond Inferring Class Representatives shows that watching one user's updates across rounds enables user-level reconstruction. That makes a persistent per-client table worse than a single intercepted update. Finally, the attack literature shows that averaging over a batch is not a shield:

All three attacks assume one gradient computed at one known model version. Our sums break that assumption, and we adapt the attacks to model it.

Other work constrains the design. Lightweight Federated Learning with Differential Privacy and Straggler Resilience (LightDP-FL) argues that secure multi-party protocols stall when participants drop out mid-round. We address this by running secure aggregation only among clients already present in a window, and we measure the cost directly. Cutting Through Privacy shows that a malicious server that crafts the model weights it sends can slice apart batches of 4096 using the first layer. Secure aggregation cannot stop that attacker, so our guarantee is scoped to honest-but-curious servers. We still report the active attack as a stated limitation, not a hidden one.

Serverless designs are the closest neighbors. ERIS splits compressed gradients into shards spread across client aggregators. It assumes synchronous rounds with no collusion and does not address stragglers. We keep a coordinator but remove its memory, which is a different trade. Peer-to-peer designs remove the server, but On the (In)security of Peer-to-Peer Decentralized Machine Learning shows that sparse peer-to-peer training leaks more than a central server, not less. That result is why we do not choose gossip as the fix. FedVS achieves straggler tolerance with privacy through coded computing, but only for vertical federated learning and only for networks with polynomial activations. Our method works with ordinary ReLU networks in horizontal federated learning.

Research Program

The work has three parts.

The first part is an attack study of MIFA as published. We simulate the unavailability patterns MIFA considers, including correlated, non-IID dropout. We then run Inverting Gradients, GradInversion, and iLRG against the stored table entries, as single snapshots and with multi-round tracking.

The second part builds client-held memory in two variants:

  • Clear variant: differences are sent unencrypted. This is an ablation that should fail through the adding-up attack.
  • Secure variant: differences go through secure aggregation with a minimum group size $k$. The server waits for $k$ arrivals, or a timeout that holds the window open, before unmasking.

The third part is a set of adaptive attacks written for the secure variant. The attacker optimizes dummy data for both the new and the old batch of every client in a window. Each dummy batch is differentiated at the model version that client actually used, and the attacker matches the signed sum. We also give the attacker a stronger position in which it knows which clients arrived and their staleness. A further attack combines consecutive windows to check whether the arrival pattern ever isolates one client. We will prove when this happens: a client is isolated exactly when some window, or some linear combination of windows, contains only that client's differences. The minimum group size rules this out by construction. The attacker's remaining lever is the timeout policy, which we report explicitly.

Baselines

The naive reference points are FedAvg that waits for everyone, and "biased FedAvg" that drops non-responders. The latter is the variant MIFA shows converging to higher training loss. Neither stores per-client vectors, but the first is slow and the second is biased. Both are run on the same data partitions, unavailability traces, and tuning budget as everything else.

Unmodified MIFA is the reference for both utility and exposure. Our secure variant must match its accuracy-versus-wall-clock curve up to the delay introduced by the minimum group size, while driving reconstruction quality down. Unmodified MIFA's leakage is expected to approach the no-defense single-client level. For that level the file reports 59.20 dB reconstruction peak signal-to-noise ratio (PSNR) on MNIST with LeNet, in Concealing Sensitive Samples. For MIFA's own accuracy numbers, no reported result was found on Lacuna, so we will reproduce MIFA ourselves.

The strongest straggler competitor is FADAS, which buffers asynchronous arrivals and shrinks the global step size when delays are large. Its reported result is about 79.7% CIFAR-10 accuracy with ResNet-18 under large delays, against about 75.3% for FedAsync. We run FADAS with secure aggregation over its buffer so that its privacy matches ours. Beating it means reaching equal or higher accuracy at equal wall-clock under correlated, non-IID unavailability, which is the regime where MIFA's unbiasedness should matter. Under benign, independent delays we expect a tie, and we will report it as one.

The privacy competitor is MIFA with per-client Gaussian noise added before storage, run in the spirit of LightDP-FL. We match reconstruction quality and compare accuracy.

The ablations are chosen to isolate the key ingredient. The clear-difference variant tests whether secure aggregation is necessary. Secure aggregation with $k=1$, meaning unmask whenever anyone arrives, tests whether the minimum group size is necessary.

Why not just…?

Why not just add differential privacy noise to MIFA's stored updates? Remembered updates are reused every round, but the noise is drawn only once per update, so it gives little averaging benefit. Local noise large enough to stop direction matching also costs heavily. LightDP-FL reports vanilla local-noise baselines more than 15% below structured schemes on CIFAR-10, and iLRG reports that differential privacy barely affects label recovery on untrained models. We still run this baseline at matched reconstruction quality. If it matches our accuracy there, the case for our cryptographic approach weakens, and we will say so.

Why not just use buffered asynchronous training such as FADAS with secure aggregation? This is the real threat. It is private and tolerates stragglers, but it lacks MIFA's guarantee under arbitrary unavailability. Its buffers also face the same small-window leakage we analyze. The FADAS baseline decides this question empirically.

Why not just let the server keep a running sum and discard the table? The server cannot remove a client's old contribution from the sum without knowing that contribution. Either the server remembers it, which recreates the honeypot, or the client does. Client-held memory is the minimal version of this idea, and it adds only $O(d)$ storage per client.

Why not just rely on large local batches? Direction 1 already rules this out. iLRG and the hyperplane attack succeed at batch size 4096.

Proposed Experiments

We train on CIFAR-10 and CIFAR-100 with ResNet-18 under MIFA-style and FADAS-style unavailability traces. We run attacks on MNIST with LeNet, on CIFAR-100, and on ImageNet with ResNet-50 at window sizes matched to GradInversion's batch range of 8 to 48. The metrics are:

  • reconstruction PSNR, SSIM, and LPIPS;
  • identifiability precision;
  • iLRG label-count accuracy;
  • test accuracy against wall-clock time;
  • server memory.

The decisive checks are these. The clear ablation should reach near no-defense PSNR through the adding-up attack. Unmodified MIFA's table should approach the same level. For the secure variant, we trace the curve of attack quality against $k$. The central result is the smallest $k$ at which LPIPS on ImageNet is clearly worse than GradInversion's reported 0.484 on synchronous batches of comparable size. That comparison tells us whether mixing model versions and signs genuinely hurts inversion or whether version-aware attacks erase the effect. The final check is the wall-clock cost of that $k$ against FADAS with a secure buffer.