Source Inference Attacks: Beyond Membership Inference Attacks in Federated Learning

Hongsheng Hu, Xuyun Zhang, Zoran Salcic, Lichao Sun, Kim-Kwang Raymond Choo, Gillian Dobbie

TL;DR

While Federated Learning (FL) is designed to keep data private, this paper introduces a potent new vulnerability: the Source Inference Attack (SIA). Beyond just knowing if a record was used for training, a curious server can now pinpoint exactly which client provided it, potentially leading to discrimination or targeted privacy breaches in sensitive fields like healthcare.

What Problem Does This Paper Address?

Federated Learning is often marketed as a "privacy-by-design" solution because raw data never leaves the client's device. Instead, clients only share model updates (like gradients or weights). However, researchers have already shown that these updates can leak information through Membership Inference Attacks (MIAs), which determine if a specific record was part of the training set.

The authors identify a critical gap: Source Privacy. In a scenario where multiple hospitals collaboratively train a model, an MIA might reveal that a specific patient's record was used. An SIA goes further, identifying which hospital provided that record. If one hospital specializes in high-risk or stigmatized conditions, identifying the source hospital can lead to discrimination against the patient, even if their raw data remains "hidden."

Methods

The paper proposes a novel SIA framework designed for an honest-but-curious server. This means the server follows the FL protocol perfectly (it doesn't "cheat" or send malicious models) but passively analyzes the updates it receives.

The core intuition is rooted in Bayesian theory: a model will typically have a much lower prediction loss on the data it was trained on compared to data it hasn't seen. The authors demonstrate that by calculating the prediction loss of each client's local model on a target record, the server can infer the source: the client with the smallest loss is the most likely owner.

The authors adapt this attack for three major FL frameworks:

  • FedSGD-SIA: The server uses received gradients to reconstruct the local models for each client.
  • FedAvg-SIA: The server directly uses the local model weights uploaded by clients.
  • FedMD-SIA: In this heterogeneous framework, the server uses "student models" to mimic the behavior of local models based on their predictions on a public dataset.
An overview of how an honest-but-curious server implements SIAs in Federated Learning.

Datasets

The researchers evaluated their attacks on a diverse set of benchmarks:

  • Synthetic: An IID dataset used to precisely manipulate data heterogeneity.
  • MNIST / FEMNIST: Standard and federated versions of handwritten digit datasets.
  • CIFAR-10 / CIFAR-100: Popular image classification datasets.
  • Purchase: A tabular dataset based on customer shopping history.

To simulate real-world conditions, they used a Dirichlet distribution ($\alpha$) to control data heterogeneity (non-IIDness), allowing them to test how the attack performs when clients have very different types of data.

Key Insights and Ideas

  • Overfitting is the "Smoking Gun": The primary reason SIAs succeed is the overfitting of local models. When a local model "memorizes" its private training data too well, it leaves a distinct fingerprint that the server can detect.
  • Heterogeneity Fuels the Attack: The more "non-IID" the data is (i.e., the more unique each client's dataset is), the more successful the attack becomes. If a client only sees "cat" images, its model becomes exceptionally good at recognizing its own cats but poor at recognizing others, making its loss profile highly distinguishable.
  • Passive but Potent: Because the attack is non-intrusive and follows the standard FL protocol, it is incredibly difficult for clients to detect that their source privacy is being compromised.
  • Training Duration Matters: Generally, the more local epochs a client runs before updating the server, the more the local model overfits, and the higher the attack success rate.

Key Results

The experimental results confirm that SIAs are a serious threat:

  • High Success Rates: In a setup with 10 clients (where random guessing is 10%), the SIA achieved a success rate of 67.1% on the Purchase dataset and over 50% on CIFAR-10 when using the FedAvg framework.
  • Impact of Data Distribution: On CIFAR-10, the attack success rate jumped from ~17% to 58.3% as the data became more heterogeneous (moving from $\alpha=100$ to $\alpha=0.1$).
  • Defense Failure: The authors tested Differential Privacy (DP) as a defense. While DP did reduce the attack's success, it caused a "significant model utility reduction." In many cases, the global model's accuracy dropped so low that the federated training became useless.

Limitations and Future Work

The authors acknowledge several areas for further exploration:

  • Scalability: The experiments focused on 10 clients. In "cross-device" FL with thousands or millions of clients (like mobile phones), the attack may be significantly harder to execute successfully.
  • Model Complexity: The study used relatively small CNN and MLP architectures. Investigating how massive models like ResNet or Transformers affect source leakage is a logical next step.
  • Defense Gaps: Since standard DP isn't a perfect solution due to the utility trade-off, there is an urgent need for new regularization techniques or FL-specific protocols that protect source privacy without destroying model performance.
  • Vertical FL: This paper focused on horizontal FL (same features, different samples). Investigating source inference in vertical FL (different features for the same samples) remains an open challenge.

Summary updated

Abstract

Federated learning (FL) is a popular approach to facilitate privacy-aware machine learning since it allows multiple clients to collaboratively train a global model without granting others access to their private data. It is, however, known that FL can be vulnerable to membership inference attacks (MIAs) , where the training records of the global model can be distinguished from the testing records. Surprisingly, research focusing on the investigation of the source inference problem appears to be lacking. We also observe that identifying a training record's source client can result in privacy breaches extending beyond MIAs. Seeking to contribute to the literature gap, we take the first step to investigate source privacy in FL. Specifically, we propose a new inference attack (hereafter referred to as source inference attack–SIA), designed to facilitate an honest-but-curious server to identify the training record's source client. The proposed SIAs leverage the Bayesian theorem to implement the attack in a non-intrusive manner without deviating from the defined FL protocol. We then evaluate SIAs in three different FL frameworks to show that in existing FL frameworks, the clients sharing gradients, model parameters, or predictions on a public dataset will leak such source information to the server. The experimental results validate the efficacy of the proposed SIAs, e.g., an attack success rate of 67.1% (baseline 10%) can be achieved when the clients share model parameters with the server. Comprehensive ablation studies demonstrate that the success of an SIA is directly related to the overfitting of the local models.
BibTeX
@article{hu2024source,
  author = {Hongsheng Hu and Xuyun Zhang and Zoran Salcic and Lichao Sun and Kim-Kwang Raymond Choo and Gillian Dobbie},
  title = {{Source Inference Attacks: Beyond Membership Inference Attacks in Federated Learning}},
  journal = {IEEE Trans. Dependable Secur. Comput.},
  year = {2024},
  doi = {10.1109/tdsc.2023.3321565},
}

Loading related content...

Figure