Subject Membership Inference Attacks in Federated Learning

Anshuman Suri, Pallika Kanani, Virendra J. Marathe, Daniel W. Peterson

Introduction: The Subject-Level Privacy Gap

In the landscape of privacy-preserving machine learning, distinctions are typically made between item-level privacy (protecting a single data point) and user-level privacy (protecting an entire participant's contribution). However, in cross-silo Federated Learning (FL), a single individual's data—a "subject"—may be distributed across multiple participating organizations. For instance, a patient's medical records might be split between different hospitals.

Existing membership inference attacks often assume the attacker has access to the exact records used for training. This research addresses a more realistic scenario: an attacker who possesses representative data (e.g., a few different photos of a target's face) and seeks to infer if that subject's data was included in the federation.

Figure 1: Conceptual overview of subject-level membership inference. Data from subjects (S0-S3) is distributed across multiple users. An adversary attempts to infer the presence of a specific subject's data.

Attack Methodology

The authors propose two black-box attacks based on the observation that models typically exhibit lower loss on data from subjects seen during training.

  1. Loss-Threshold Attack: The adversary evaluates the model's loss on representative samples of a target subject. If the loss falls below a calibrated threshold, the subject is classified as a member.
  2. Loss-Across-Rounds Attack: Specific to FL, this attack monitors the model's loss on a target subject across successive training rounds. A consistent decrease in loss suggests the subject's data is actively influencing the model's parameters.

These attacks are practical, requiring only a small "shadow" set of 5–10 known subjects to calibrate thresholds.

Experimental Results

The study evaluated these attacks across various datasets, including FEMNIST (handwritten digits), Shakespeare (text), and a synthetic data generator. On the FEMNIST dataset, the attacks proved remarkably potent, achieving F1 scores between 0.85 and 0.95.

Crucially, the results demonstrated that the architectural separation of Federated Learning does not inherently provide mathematical privacy. As shown in the figure below, the attack success on a federated model increases over training rounds, eventually matching the risk level of a centrally trained "combined" model.

Figure 2: Attack F-1 score across training rounds. The federated model's vulnerability increases over time, approaching the risk level of a centrally combined dataset.

Key Insights and the DP Trade-off

  • The Power of Representative Data: Privacy can be compromised even without the exact training pixels. A few non-training samples are sufficient for high-accuracy inference.
  • Dimensionality and Overfitting: Higher-dimensional data increases subject uniqueness, making them easier to identify. Interestingly, these attacks are effective even in early training stages, suggesting that overfitting is not the sole driver of vulnerability.
  • The Differential Privacy (DP) Cost: While DP can mitigate these attacks, it comes at a high cost to utility. To bring attack success down to baseline levels, model accuracy often plummeted (e.g., from 92% to 63%), highlighting a significant challenge for practical deployment.

Conclusion

Subject-level membership inference represents a significant threat in Federated Learning. The ability to execute these attacks with only representative data and minimal prior knowledge makes them a realistic concern for cross-silo applications. Future research must focus on developing defense mechanisms that can protect subject privacy without the prohibitive utility losses associated with current techniques.

Summary updated

Abstract

Privacy attacks on Machine Learning (ML) models often focus on inferring the existence of particular data points in the training data. However, what the adversary really wants to know is if a particular individual's (subject's) data was included during training. In such scenarios, the adversary is more likely to have access to the distribution of a particular subject than actual records. Furthermore, in settings like cross-silo Federated Learning (FL), a subject's data can be embodied by multiple data records that are spread across multiple organizations. Nearly all of the existing private FL literature is dedicated to studying privacy at two granularities -- item-level (individual data records), and user-level (participating user in the federation), neither of which apply to data subjects in cross-silo FL. This insight motivates us to shift our attention from the privacy of data records to the privacy of data subjects, also known as subject-level privacy. We propose two novel black-box attacks for subject membership inference, of which one assumes access to a model after each training round. Using these attacks, we estimate subject membership inference risk on real-world data for single-party models as well as FL scenarios. We find our attacks to be extremely potent, even without access to exact training records, and using the knowledge of membership for a handful of subjects. To better understand the various factors that may influence subject privacy risk in cross-silo FL settings, we systematically generate several hundred synthetic federation configurations, varying properties of the data, model design and training, and the federation itself. Finally, we investigate the effectiveness of Differential Privacy in mitigating this threat.
BibTeX
@misc{suri2022subject,
  author = {Anshuman Suri and Pallika Kanani and Virendra J. Marathe and Daniel W. Peterson},
  title = {{Subject Membership Inference Attacks in Federated Learning}},
  year = {2022},
  eprint = {2206.03317},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2206.03317},
}
Figure