Understanding Practical Membership Privacy of Deep Learning

Marlon Tobaben, Gauri Pradhan, Yuan He, Joonas Jälkö, Antti Honkela

Abstract

We apply a state-of-the-art membership inference attack (MIA) to systematically test the practical privacy vulnerability of fine-tuning large image classification models. We focus on understanding the properties of data sets and samples that make them vulnerable to membership inference. In terms of data set properties, we find a strong power law dependence between the number of examples per class in the data and the MIA vulnerability, as measured by true positive rate of the attack at a low false positive rate. We train a linear model to predict true positive rate based on data set properties and observe good fit for MIA vulnerability on unseen data. To analyse the phenomenon theoretically, we reproduce the result on a simplified model of membership inference that behaves similarly to our experimental data. We prove that in this model, the logarithm of the difference of true and false positive rates depends linearly on the logarithm of the number of examples per class. For an individual sample, the gradient norm is predictive of its vulnerability.

Introduction: The Privacy Gap in Fine-Tuning

In modern machine learning, fine-tuning pre-trained models on specialized datasets is a standard practice. However, this raises significant concerns regarding Membership Inference Attacks (MIAs), where an adversary attempts to determine if a specific data point was used during training. While Differential Privacy (DP) offers formal mathematical guarantees, its bounds are often perceived as overly conservative. Conversely, empirical MIAs provide a lower bound on risk but lack a predictive framework for how dataset characteristics influence vulnerability. This research bridges that gap by establishing a predictable relationship between dataset properties and privacy risk.

Methodology

The study employs a dual approach:

  1. Theoretical Modeling: The authors developed a simplified mathematical model of a "Head" classifier. By analyzing data points as vectors in high-dimensional space, they derived a power-law relationship: $\log(TPR - FPR) \approx -\beta_S \log(S) - \beta_0$, where $S$ represents the number of samples per class (shots).
  2. Empirical Validation: These theoretical findings were tested using state-of-the-art attacks, specifically LiRA (Likelihood Ratio Attack) and RMIA (Robust Membership Inference Attack). The experiments involved fine-tuning Vision Transformers (ViT-B) and ResNets on a subset of the VTAB benchmark, including datasets like Patch Camelyon, EuroSAT, and CIFAR-100.

The Power Law of Privacy Risk

The central finding is that MIA vulnerability scales predictably with dataset size. As the number of examples per class ($S$) increases, the attacker's advantage (the difference between True Positive Rate and False Positive Rate) decays following a consistent power law across diverse data domains.

MIA vulnerability (TPR-FPR) follows a power-law decay as the number of examples per class increases across various datasets.

While the number of classes ($C$) also influences vulnerability, its impact is secondary to the density of samples per class. This relationship allows for the prediction of privacy risk based on simple dataset metrics.

Predictive Modeling and the "DP Gap"

The researchers trained a linear regression model to predict the True Positive Rate (TPR) based on $\log(S)$ and $\log(C)$. This model achieved an impressive $R^2$ of 0.930, demonstrating that practical privacy risk is highly predictable even on unseen data.

Observed vs. predicted TPR for the linear regression model, demonstrating high predictive accuracy across different fine-tuning regimes.

However, a critical distinction emerged between "average" and "worst-case" vulnerability. While average privacy improves rapidly with more data (slope $\beta_S \approx -0.5$), the most vulnerable "outlier" samples see their risk decrease much more slowly (slope $\beta_S \approx -0.27$).

Extrapolating this power law reveals a massive "DP Gap." To achieve a level of protection equivalent to a strong DP guarantee ($\epsilon=1$), an average sample might only require a few hundred examples per class. In contrast, protecting the most vulnerable individual samples to the same standard would require millions of samples per class—a scale far beyond most practical fine-tuning scenarios.

Conclusion and Limitations

This work provides a quantitative framework for understanding membership privacy in transfer learning. While the power law offers a tool for risk assessment, it also highlights the difficulty of protecting outliers without formal DP.

Current limitations include the focus on "Head" fine-tuning; training from scratch or full-model fine-tuning may lead to higher memorization. Additionally, the model assumes relatively well-behaved data distributions. Future research will explore how data imbalance and adversary side-knowledge might alter these fundamental privacy laws.

Summary updated

Abstract

We apply a state-of-the-art membership inference attack (MIA) to systematically test the practical privacy vulnerability of fine-tuning large image classification models. We focus on understanding the properties of data sets and samples that make them vulnerable to membership inference. In terms of data set properties, we find a strong power law dependence between the number of examples per class in the data and the MIA vulnerability, as measured by true positive rate of the attack at a low false positive rate. We train a linear model to predict true positive rate based on data set properties and observe good fit for MIA vulnerability on unseen data. To analyse the phenomenon theoretically, we reproduce the result on a simplified model of membership inference that behaves similarly to our experimental data. We prove that in this model, the logarithm of the difference of true and false positive rates depends linearly on the logarithm of the number of examples per class.For an individual sample, the gradient norm is predictive of its vulnerability.
BibTeX
@inproceedings{tobaben2024understanding,
  author = {Marlon Tobaben and Gauri Pradhan and Yuan He and Joonas Jälkö and Antti Honkela},
  title = {{Understanding Practical Membership Privacy of Deep Learning}},
  booktitle = {PML @ ICLR},
  year = {2024},
  url = {https://openreview.net/forum?id=28TUPQWhBA},
}
Figure