Privacy-preserving machine learning often relies on a comforting mathematical illusion: if an individual data point or model update is hidden inside a crowd or masked by noise, it remains secure. In Federated Learning (FL), protocols like Secure Aggregation (SecAgg) ensure that a central server only observes the sum of updates from hundreds of users, never an individual contribution. Similarly, Differential Privacy (DP) injects calibrated stochastic noise to obscure any single participant's traits.
These defenses work remarkably well—but only if you look at a single training round in isolation.
When a distributed system runs across multiple rounds, an adversary can observe how the outputs change over time. By combining these sequential observations, the adversary can systematically strip away cryptographic and statistical protections. Understanding why this multi-round leakage occurs requires looking at the system through three lenses: linear algebra, signal processing, and optimization constraints.
The Linear Algebra of Shifting Crowds
To understand how secure aggregation fails over time, imagine an adversary trying to discover a secret number $x_1$ held by Alice. In a single round, Alice mixes her number with updates from Bob ($x_2$) and Charlie ($x_3$). The server only receives the aggregate sum:
$$S_1 = x_1 + x_2 + x_3$$
Because $x_2$ and $x_3$ are unknown, Alice's secret is perfectly protected. This is the core guarantee of single-round secure aggregation.
However, distributed systems are dynamic. In real-world deployments, devices drop offline, battery levels fluctuate, and user selection algorithms randomly sample different subsets of participants at each step. Suppose that in the next round, Bob drops out, and the server collects an aggregate from only Alice and Charlie:
$$S_2 = x_1 + x_3$$
By simply subtracting the second aggregate from the first ($S_1 - S_2$), the server instantly isolates Bob's private update $x_2$.
As formalized by So et al. (2021) and Lam et al. (2021), an adversarial server can treat the history of a federated learning session as a system of linear equations:
$$Y = AX$$
Where $Y$ is the vector of observed aggregates across multiple rounds, $X$ is the matrix of private individual user updates, and $A$ is the user participation matrix tracking which devices contributed to which rounds.
While a single row of this system leaves $X$ completely underdetermined, a curious server monitoring the system over many rounds can accumulate enough unique equations to make the participation matrix $A$ invertible. Once $A$ is known or reconstructed, the server can "disaggregate" the gradients, isolating individual user updates and exposing them to high-fidelity data reconstruction attacks (Lam et al., 2021). As noted in a comprehensive survey by Liu et al. (2022), this vulnerability stems directly from inconsistent user participation; the presence or absence of a specific participant between rounds betrays their unique influence on the global model.
This intersection vulnerability is a fundamental property of distributed state optimization. Long before the rise of modern federated learning, Greenstadt (2006) demonstrated that distributed constraint optimization algorithms suffer from the exact same flaw. When independent agents repeatedly solve variations of a problem, an adversary monitoring the shifting outputs can perform an intersection attack to isolate and infer private variables that were otherwise protected during a single execution.
Averaging Out Differential Privacy
If cryptographic aggregation fails because of linear dependencies, can we rely on statistical defenses like Differential Privacy? User-Level Differential Privacy (UDP) adds random, zero-mean noise to a user's update before sending it to the server, ensuring that the presence of any single user's data cannot be confidently inferred.
The weakness here is a classic signal-processing problem: zero-mean noise cancels out when averaged over time.
If a user participates in multiple training rounds using the same underlying dataset, their underlying model updates will contain a consistent, structural signal determined by their private data. The DP noise injected in each round, however, is independent and transient. By tracking a specific user's updates across a sequence of rounds, an attacker can aggregate the noisy updates (Feng et al., 2022). This temporal accumulation acts as a low-pass filter, smoothing away the stochastic noise and sharpening the underlying sensitive attributes (such as gender or emotion in speech models).
A similar phenomenon occurs in membership inference attacks. Zhu (2025) demonstrates that while a target client's membership signal might be weak and masked by noise in any single communication round, aggregating these signals across multiple rounds filters out the transient noise. This allows an adversary to confidently determine whether a specific user's data was part of the training set, even when common privacy defenses are active.
Temporal Constraints and Iterative Refinement
Multi-round observation does more than just solve linear equations or average out noise; it provides an optimization landscape rich with mathematical constraints.
Consider a gradient inversion attack, where an adversary tries to reconstruct an input image from its calculated gradient. In a standard single-round setting, if a user processes a large batch of images, the gradients of all those images are averaged together. This averaging creates massive ambiguity—many different combinations of images can produce the exact same average gradient, making perfect reconstruction nearly impossible.
However, machine learning models typically train over multiple epochs, meaning the same data batches are processed repeatedly as the global model parameters evolve. Li et al. (2023) show that observing how a user's gradients change across multiple time steps provides additional mathematical constraints that resolve the single-step ambiguity. Instead of guessing a batch from a single snapshot, the adversary uses robust optimization to find an image batch whose gradient trajectory matches the observed temporal sequence.
Single-Round: [ Batch of Data ] ──> [ Average Gradient ] ──> (High Ambiguity for Attacker)
Multi-Round: [ Same Batch ] ──> [ Gradient Epoch 1 ] ┐
[ Same Batch ] ──> [ Gradient Epoch 2 ] ┼─> (Temporal Constraints Resolve Ambiguity)
[ Same Batch ] ──> [ Gradient Epoch 3 ] ┘
By jointly optimizing the reconstruction across overlapping epochs, attackers can significantly sharpen the recovered data (Xu et al., 2022).
This cumulative information leakage manifests across various interaction paradigms:
- Active Trapping: An adversarial server can actively exploit persistent learning sessions. Wen et al. (2022) developed a "fishing" attack where the server sends specially crafted model weights to users to artificially magnify the gradient of a target data point. By deploying binary search strategies over multiple communication rounds, the attacker can systematically isolate and reconstruct every single data point within a user's batch.
- Model Snapshots: Privacy leaks also occur when models are updated statically. Hui et al. (2023) showed that an attacker with access to model snapshots before and after data updates can infer changes to individual records, particularly when consistent changes are made across multiple records.
- Continuous Queries: The risk extends to post-deployment interfaces. Kumari et al. (2024) modeled the repeated interaction between a user and an explainable AI system as a continuous-time stochastic signaling game. They proved that an attacker can treat the evolving variance of model explanations across multiple queries as signals, iteratively "inching closer" to a perfect membership inference threshold without any prior knowledge of the model's internal state.
This reality challenges how we evaluate model repositories and learnware specifications; as Lei et al. (2026) point out, evaluating a model summary in isolation fails to account for adaptive multi-query attacks where an adversary uses a sequence of related queries to triangulate sensitive training data.
Designing for Long-Term Privacy
The recurring theme across these papers is clear: evaluating privacy based on a single snapshot creates a false sense of security. If the user selection or query mechanism is random or unconstrained, information inevitably leaks across rounds.
To mitigate this, defenses must become temporally aware. For instance, instead of allowing random user participation in every round, So et al. (2021) and So et al. (2023) introduced Multi-RoundSecAgg. This framework uses structured batch partitioning to govern user selection over time. By forcing users into fixed, pre-determined groups across training cycles, it prevents the central server from ever constructing an invertible or overdetermined system of linear equations, neutralizing the mathematical foundation of the intersection attack while preserving utility.
Key Foundational Work
Gradient Disaggregation: Breaking Privacy in Federated Learning by Reconstructing the User Participant Matrix (2021) — contributed 3 concept elements to this direction.
Privatizing Constraint Optimization (2006) — contributed 3 concept elements to this direction.
Securing Secure Aggregation: Mitigating Multi-Round Privacy Leakage in Federated Learning (2021) — contributed 2 concept elements to this direction.