Comprehensive Privacy Analysis of Deep Learning: Stand-alone and Federated Learning under Passive and Active White-box Inference Attacks

Milad Nasr, Reza Shokri, Amir Houmansadr

Abstract: Deep neural networks are susceptible to various inference attacks as they remember information about their training data. We design white-box inference attacks to perform a comprehensive privacy analysis of deep learning models. We measure the privacy leakage through parameters of fully trained models as well as the parameter updates of models during training. We design inference algorithms for both centralized and federated learning, with respect to passive and active inference attackers, and assuming different adversary prior knowledge. We evaluate our novel white-box membership inference attacks against deep learning algorithms to trace their training data records. We show that a straightforward extension of the known black-box attacks to the white-box setting (through analyzing the outputs of activation functions) is ineffective. We therefore design new algorithms tailored to the white-box setting by exploiting the privacy vulnerabilities of the stochastic gradient descent algorithm, which is the algorithm used to train deep neural networks. We investigate the reasons why deep learning models may leak information about their training data. We then show that even well-generalized models are significantly susceptible to white-box membership inference attacks, by analyzing state-of-the-art pre-trained and publicly available models for the CIFAR dataset. We also show how adversarial participants, in the federated learning setting, can successfully run active membership inference attacks against other participants, even when the global model achieves high prediction accuracies.

Introduction

Deep learning models are far more "leaky" than previously thought. This research introduces white-box membership inference attacks (MIA), demonstrating that even well-generalized models—those that perform perfectly on unseen data—leave a distinct "footprint" of their training sets. By exploiting how Stochastic Gradient Descent (SGD) pushes gradients of training data toward zero, attackers can compromise user privacy by determining if a specific record was used during training.

While prior research focused on "black-box" attacks (observing only final predictions), modern collaborative environments like Federated Learning expose internal model parameters and gradients. This paper addresses the critical privacy risks in these white-box settings, exploring both passive observation and active manipulation of the training process.

Methodology

The proposed white-box attack framework extracts features from multiple internal sources to distinguish members from non-members. The researchers found that simply extending black-box methods to analyze activation functions is ineffective; instead, they developed algorithms that exploit the specific vulnerabilities of the SGD training process:

  1. Gradient Outputs: The core innovation. SGD minimizes loss by pushing training sample gradients toward zero, making "member" gradients look fundamentally different from "non-member" ones.
  2. Intermediate Activations: The outputs of hidden layers.
  3. Loss and Predictions: Standard metrics used in traditional black-box attacks.
White-box access to internal model parameters and activations.

Attack Variations

  • Passive vs. Active: Passive attackers observe the model. Active attackers in federated settings perform gradient ascent on target data before uploading updates, forcing the global model to react more visibly to specific points.
  • Supervised vs. Unsupervised: When labeled "shadow datasets" are unavailable, the researchers use an encoder-decoder architecture to find low-dimensional embeddings of membership information, followed by clustering to separate members from non-members.

Evaluation and Key Results

The attacks were evaluated across three domains: image recognition (CIFAR100), shopping history (Purchase100), and hospital discharge data (Texas100), using architectures like AlexNet, ResNet, and DenseNet.

Generalization is Not a Privacy Shield

A common misconception is that high generalization (low overfitting) protects privacy. This study proves that even models with high test accuracy are susceptible to white-box attacks because gradients leak information that predictions do not.

Vulnerabilities in Federated Learning

In Federated Learning (FL), participants exchange model updates with a central aggregator. This exchange is a primary source of leakage, as malicious participants can observe or manipulate the global parameter updates.

Federated learning architecture where participants exchange model parameters.
  • White-box vs. Black-box Performance: On a DenseNet model trained on CIFAR100, the white-box attack achieved 74.3% accuracy, compared to just 54.5% for standard black-box methods.
  • Active Attack Devastation: A malicious participant in FL can "isolate" others or inject malicious gradients, boosting attack accuracy from ~70% to over 92%.
  • Layer-wise Leakage: Later layers of a neural network leak significantly more information than early layers, as they specialize in specific relationships found in the training set.

Conclusion

The Stochastic Gradient Descent algorithm is the primary source of these privacy vulnerabilities. While Differential Privacy (DP) can mitigate these attacks by adding noise to gradients, it often reduces model utility. Future work must establish theoretical lower bounds for privacy leakage and develop defenses that protect gradients without sacrificing learning performance on complex tasks.

Summary updated

Abstract

Deep neural networks are susceptible to various inference attacks as they remember information about their training data. We design white-box inference attacks to perform a comprehensive privacy analysis of deep learning models. We measure the privacy leakage through parameters of fully trained models as well as the parameter updates of models during training. We design inference algorithms for both centralized and federated learning, with respect to passive and active inference attackers, and assuming different adversary prior knowledge. We evaluate our novel white-box membership inference attacks against deep learning algorithms to trace their training data records. We show that a straightforward extension of the known black-box attacks to the white-box setting (through analyzing the outputs of activation functions) is ineffective. We therefore design new algorithms tailored to the white-box setting by exploiting the privacy vulnerabilities of the stochastic gradient descent algorithm, which is the algorithm used to train deep neural networks. We investigate the reasons why deep learning models may leak information about their training data. We then show that even well-generalized models are significantly susceptible to white-box membership inference attacks, by analyzing state-of-the-art pre-trained and publicly available models for the CIFAR dataset. We also show how adversarial participants, in the federated learning setting, can successfully run active membership inference attacks against other participants, even when the global model achieves high prediction accuracies.
BibTeX
@inproceedings{nasr2019comprehensive,
  author = {Milad Nasr and Reza Shokri and Amir Houmansadr},
  title = {{Comprehensive Privacy Analysis of Deep Learning: Stand-alone and Federated Learning under Passive and Active White-box Inference Attacks}},
  booktitle = {IEEE Symposium on Security and Privacy},
  year = {2019},
  url = {https://openreview.net/forum?id=meJ5A_sjXh},
}
Figure