Introduction: The Subject-Level Privacy Gap
In the landscape of privacy-preserving machine learning, distinctions are typically made between item-level privacy (protecting a single data point) and user-level privacy (protecting an entire participant's contribution). However, in cross-silo Federated Learning (FL), a single individual's data—a "subject"—may be distributed across multiple participating organizations. For instance, a patient's medical records might be split between different hospitals.
Existing membership inference attacks often assume the attacker has access to the exact records used for training. This research addresses a more realistic scenario: an attacker who possesses representative data (e.g., a few different photos of a target's face) and seeks to infer if that subject's data was included in the federation.
Attack Methodology
The authors propose two black-box attacks based on the observation that models typically exhibit lower loss on data from subjects seen during training.
- Loss-Threshold Attack: The adversary evaluates the model's loss on representative samples of a target subject. If the loss falls below a calibrated threshold, the subject is classified as a member.
- Loss-Across-Rounds Attack: Specific to FL, this attack monitors the model's loss on a target subject across successive training rounds. A consistent decrease in loss suggests the subject's data is actively influencing the model's parameters.
These attacks are practical, requiring only a small "shadow" set of 5–10 known subjects to calibrate thresholds.
Experimental Results
The study evaluated these attacks across various datasets, including FEMNIST (handwritten digits), Shakespeare (text), and a synthetic data generator. On the FEMNIST dataset, the attacks proved remarkably potent, achieving F1 scores between 0.85 and 0.95.
Crucially, the results demonstrated that the architectural separation of Federated Learning does not inherently provide mathematical privacy. As shown in the figure below, the attack success on a federated model increases over training rounds, eventually matching the risk level of a centrally trained "combined" model.
Key Insights and the DP Trade-off
- The Power of Representative Data: Privacy can be compromised even without the exact training pixels. A few non-training samples are sufficient for high-accuracy inference.
- Dimensionality and Overfitting: Higher-dimensional data increases subject uniqueness, making them easier to identify. Interestingly, these attacks are effective even in early training stages, suggesting that overfitting is not the sole driver of vulnerability.
- The Differential Privacy (DP) Cost: While DP can mitigate these attacks, it comes at a high cost to utility. To bring attack success down to baseline levels, model accuracy often plummeted (e.g., from 92% to 63%), highlighting a significant challenge for practical deployment.
Conclusion
Subject-level membership inference represents a significant threat in Federated Learning. The ability to execute these attacks with only representative data and minimal prior knowledge makes them a realistic concern for cross-silo applications. Future research must focus on developing defense mechanisms that can protect subject privacy without the prohibitive utility losses associated with current techniques.
Summary updated