Stolen Memories: Leveraging Model Memorization for Calibrated White-Box Membership Inference

Klas Leino, Matt Fredrikson

This paper, "Stolen Memories," by Klas Leino and Matt Fredrikson from Carnegie Mellon University, presents a powerful new approach to Membership Inference (MI) attacks in a white-box setting. It demonstrates that even when a model appears to generalize well, its internal weights still leak significant information about its training data.

TL;DR

The authors develop a white-box membership inference attack that exploits how models "memorize" idiosyncratic features of training data, even when they don't show obvious signs of overfitting. Their method can be precisely calibrated for high-confidence inferences and reveals that common defenses, like large-$\epsilon$ differential privacy, are often surprisingly ineffective.

What Problem Does This Paper Address?

Membership Inference (MI) attacks aim to determine if a specific data point was part of a model's training set. This is a major privacy concern, especially in sensitive domains like healthcare or finance.

Most existing MI attacks are "black-box," meaning the attacker only sees the model's final outputs (like class probabilities). While effective, these attacks are limited. Previous "white-box" attacks (where the attacker has access to the model's internal weights and activations) often required the attacker to already know a large portion of the training data to be effective.

This paper tackles the challenge of mounting a powerful white-box MI attack without requiring prior access to the target model's training data, by looking deeper into how neural networks actually learn—and overfit—features.

Methods

The core intuition is that an overfit model uses features in "idiosyncratic" ways. For example, if a face recognition model correctly identifies "Tony Blair" not just by his face, but because one specific training image had a unique pink background, that background becomes "evidence" of membership.

Figure 1: Idiosyncratic feature use. The model uses the face for test instances (b) but relies on a unique pink background for a training instance (c).

The authors formalize this through several steps:

  1. Bayes-Optimal Attack: They theoretically derive an optimal attack for a linear model assuming Gaussian data distributions. This shows that membership can be inferred by comparing the empirical training distribution to the general population distribution. Figure 2: Theoretical Gaussian distributions showing how a point x' is more likely to come from the training distribution (theta-hat) than the general population (theta-star).

  2. Proxy Models (bayes-wb): Since attackers don't know the true distributions, they train "proxy models" on a separate hold-out dataset. By comparing the weights of the target model to these proxy models, they can identify the "displacement" caused by memorizing the training set.

  3. Generalized Attack (general-wb): They replace simple weight subtraction with a learned "displacement function" (a small neural network) that can identify more complex patterns of memorization. Figure 3: The generalized attack model architecture, using a learned displacement function (df) to compare target weights (W-hat) and proxy weights (W-tilde).

  4. Deep Model Extension: To handle deep networks, they use internal influence measures to create local linear approximations of each layer. They then use a "meta-model" to combine membership signals from every layer of the network.

  5. Calibration: They introduce a heuristic to calibrate the attack's confidence scores, allowing an attacker to set a threshold for high-precision inferences (e.g., "I am 95% sure this point was in the training set").
Figure 2: Theoretical Gaussian distributions showing how a point x' is more likely to come from the training distribution (theta-hat) than the general population (theta-star).
Figure 3: The generalized attack model architecture, using a learned displacement function (df) to compare target weights (W-hat) and proxy weights (W-tilde).

Datasets

The authors evaluated their attacks on a wide range of datasets:

  • Synthetic Data: Gaussian-distributed data for theoretical validation.
  • Sensitive Real-World Data: Adult (census), Pima Diabetes, Breast Cancer Wisconsin, Hepatitis, and German Credit.
  • Standard Benchmarks: MNIST, LFW (Labeled Faces in the Wild), CIFAR-10, and CIFAR-100.

Key Insights and Ideas

  • Overfitting is Internal: A model can have a very small "generalization gap" (the difference between training and test accuracy) but still exhibit significant idiosyncratic feature use internally. This means standard metrics for overfitting are poor indicators of privacy risk.
  • Every Layer Leaks: The "meta-model" approach showed that information leakage isn't confined to the final layers; representations learned throughout the entire network contribute to the attack's success.
  • Precision Matters More than Recall: For a practical privacy violation, an attacker only needs to be very sure about a few points, rather than moderately sure about all of them. The authors' calibration method makes this possible.
  • The False Security of Large-$\epsilon$ DP: Differential Privacy (DP) is the gold standard for defense, but the "privacy budget" ($\epsilon$) is often set high in practice (e.g., $\epsilon=16$) to maintain model accuracy. The authors found that at these levels, DP provides almost no protection against their attack.

Key Results

  • Outperforming Black-Box: The white-box attacks (bayes-wb and general-wb) consistently outperformed standard black-box shadow model attacks in both accuracy and, more significantly, precision.
  • High Precision on "Generalizing" Models: On MNIST, which had only a 1.1% generalization error, the attack could still be calibrated to identify a subset of training members with 75% precision.
  • Defense Failures: On several datasets (BCW, PD, LFW), training with 16-DP provided less protection than simple regularization (like Dropout), while significantly harming the model's primary accuracy.

Limitations and Future Work

  • White-Box Requirement: The attack requires full access to model weights, which may not always be available to an adversary (though the authors argue this is increasingly common via "Model Zoos" or security breaches).
  • Defense Research: The results highlight a dire need for better defenses. The authors suggest that transfer learning (pre-training on public data and only fine-tuning the top layers on private data) might be a more practical defense than standard DP, as it limits the number of layers that can leak sensitive information.
  • Hyperparameter Tuning: While the general-wb attack is robust, it does introduce new hyperparameters (like the architecture of the displacement function) that require tuning via shadow models.

Summary updated

Abstract

Membership inference (MI) attacks exploit the fact that machine learning algorithms sometimes leak information about their training data through the learned model. In this work, we study membership inference in the white-box setting in order to exploit the internals of a model, which have not been effectively utilized by previous work. Leveraging new insights about how overfitting occurs in deep neural networks, we show how a model's idiosyncratic use of features can provide evidence for membership to white-box attackers---even when the model's black-box behavior appears to generalize well---and demonstrate that this attack outperforms prior black-box methods. Taking the position that an effective attack should have the ability to provide confident positive inferences, we find that previous attacks do not often provide a meaningful basis for confidently inferring membership, whereas our attack can be effectively calibrated for high precision. Finally, we examine popular defenses against MI attacks, finding that (1) smaller generalization error is not sufficient to prevent attacks on real models, and (2) while small-$ε$-differential privacy reduces the attack's effectiveness, this often comes at a significant cost to the model's accuracy; and for larger $ε$ that are sometimes used in practice (e.g., $ε=16$), the attack can achieve nearly the same accuracy as on the unprotected model.
BibTeX
@inproceedings{leino2020stolen,
  author = {Klas Leino and Matt Fredrikson},
  title = {{Stolen Memories: Leveraging Model Memorization for Calibrated White-Box Membership Inference}},
  booktitle = {USENIX Security},
  year = {2020},
}
Figure