Abstract
Federated learning (FL) is a popular approach to facilitate privacy-aware machine learning since it allows multiple clients to collaboratively train a global model without granting others access to their private data. It is, however, known that FL can be vulnerable to membership inference attacks (MIAs), where the training records of the global model can be distinguished from the testing records. Surprisingly, research focusing on the investigation of the source inference problem appears to be lacking. We also observe that identifying a training record's source client can result in privacy breaches extending beyond MIAs. Seeking to contribute to the literature gap, we take the first step to investigate source privacy in FL. Specifically, we propose a new inference attack (hereafter referred to as source inference attack — SIA), designed to facilitate an honest-but-curious server to identify the training record's source client. The proposed SIAs leverage the Bayesian theorem to allow the server to implement the attack in a non-intrusive manner without deviating from the defined FL protocol. The experimental results validate the efficacy of the proposed SIAs.
Introduction
Federated Learning (FL) is often championed as a "privacy-by-design" framework because it allows decentralized clients to train a shared model without ever exchanging raw data. Instead, clients only share model updates, such as gradients or weights. However, the privacy guarantees of FL are not absolute. While Membership Inference Attacks (MIAs) have demonstrated that an adversary can determine if a specific record was part of a training set, this paper identifies a deeper vulnerability: Source Privacy.
In sensitive applications like healthcare, knowing that a patient's record was used in training (MIA) is a breach, but identifying which hospital provided that record (SIA) can lead to even more targeted discrimination. For instance, if a specific hospital specializes in high-risk conditions, the source identification itself reveals sensitive metadata about the patient. This research introduces the Source Inference Attack (SIA) to explore this critical gap.
Methodology
The paper proposes an SIA framework tailored for an honest-but-curious server. In this scenario, the server adheres strictly to the FL protocol but passively analyzes the updates it receives to infer the source of specific data records.
The attack is grounded in Bayesian theory, utilizing the observation that a model typically exhibits a significantly lower prediction loss on its own training data than on unseen data. By calculating the prediction loss of each client's local model on a target record, the server can identify the source; the client whose model produces the smallest loss is the most likely owner. The authors adapt this approach for three major FL frameworks:
- FedSGD-SIA: The server uses received gradients to reconstruct local models for each client.
- FedAvg-SIA: The server directly utilizes the local model weights uploaded by clients.
- FedMD-SIA: In heterogeneous settings, the server employs "student models" to mimic local model behavior based on their predictions on a public dataset.
Experimental Setup
The researchers evaluated SIAs across a variety of benchmarks, including Synthetic datasets, MNIST/FEMNIST, CIFAR-10/100, and the tabular Purchase dataset. To simulate real-world conditions, they used a Dirichlet distribution ($\alpha$) to control data heterogeneity (non-IIDness). This allowed them to test how the attack performs when clients possess distinct data distributions.
Key Insights and Results
The study highlights several factors that facilitate successful source inference:
- Overfitting as a Vulnerability: The primary driver of SIAs is the overfitting of local models. When a model "memorizes" its local training data, it leaves a detectable fingerprint that the server can exploit.
- Heterogeneity Fuels the Attack: The more unique a client's data is (high non-IIDness), the more distinguishable its model's loss profile becomes. On CIFAR-10, the attack success rate jumped from ~17% to 58.3% as heterogeneity increased (moving from $\alpha=100$ to $\alpha=0.1$).
- Training Duration: Increased local training epochs lead to higher overfitting, which directly correlates with higher attack success rates.
- Defense Limitations: While Differential Privacy (DP) can mitigate SIAs, it often results in a significant reduction in model utility. In many tests, the noise required to stop the attack made the global model virtually useless for its intended task.
Overall, the results confirm that SIAs are a potent threat. In a 10-client setup, the SIA achieved a success rate of 67.1% on the Purchase dataset and over 50% on CIFAR-10 under the FedAvg framework.
Conclusion and Future Work
This work demonstrates that source privacy is a significant and overlooked risk in Federated Learning. Because SIAs are non-intrusive and follow standard protocols, they are exceptionally difficult for clients to detect. Future research is needed to address the scalability of these attacks in cross-device FL (with millions of clients), the impact of more complex architectures like Transformers, and the development of new regularization techniques that can protect source privacy without sacrificing model performance.
Summary updated