Abstract
We apply a state-of-the-art membership inference attack (MIA) to systematically test the practical privacy vulnerability of fine-tuning large image classification models. We focus on understanding the properties of data sets and samples that make them vulnerable to membership inference. In terms of data set properties, we find a strong power law dependence between the number of examples per class in the data and the MIA vulnerability, as measured by true positive rate of the attack at a low false positive rate. We train a linear model to predict true positive rate based on data set properties and observe good fit for MIA vulnerability on unseen data. To analyse the phenomenon theoretically, we reproduce the result on a simplified model of membership inference that behaves similarly to our experimental data. We prove that in this model, the logarithm of the difference of true and false positive rates depends linearly on the logarithm of the number of examples per class. For an individual sample, the gradient norm is predictive of its vulnerability.
Introduction: The Privacy Gap in Fine-Tuning
In modern machine learning, fine-tuning pre-trained models on specialized datasets is a standard practice. However, this raises significant concerns regarding Membership Inference Attacks (MIAs), where an adversary attempts to determine if a specific data point was used during training. While Differential Privacy (DP) offers formal mathematical guarantees, its bounds are often perceived as overly conservative. Conversely, empirical MIAs provide a lower bound on risk but lack a predictive framework for how dataset characteristics influence vulnerability. This research bridges that gap by establishing a predictable relationship between dataset properties and privacy risk.
Methodology
The study employs a dual approach:
- Theoretical Modeling: The authors developed a simplified mathematical model of a "Head" classifier. By analyzing data points as vectors in high-dimensional space, they derived a power-law relationship: $\log(TPR - FPR) \approx -\beta_S \log(S) - \beta_0$, where $S$ represents the number of samples per class (shots).
- Empirical Validation: These theoretical findings were tested using state-of-the-art attacks, specifically LiRA (Likelihood Ratio Attack) and RMIA (Robust Membership Inference Attack). The experiments involved fine-tuning Vision Transformers (ViT-B) and ResNets on a subset of the VTAB benchmark, including datasets like Patch Camelyon, EuroSAT, and CIFAR-100.
The Power Law of Privacy Risk
The central finding is that MIA vulnerability scales predictably with dataset size. As the number of examples per class ($S$) increases, the attacker's advantage (the difference between True Positive Rate and False Positive Rate) decays following a consistent power law across diverse data domains.
While the number of classes ($C$) also influences vulnerability, its impact is secondary to the density of samples per class. This relationship allows for the prediction of privacy risk based on simple dataset metrics.
Predictive Modeling and the "DP Gap"
The researchers trained a linear regression model to predict the True Positive Rate (TPR) based on $\log(S)$ and $\log(C)$. This model achieved an impressive $R^2$ of 0.930, demonstrating that practical privacy risk is highly predictable even on unseen data.
However, a critical distinction emerged between "average" and "worst-case" vulnerability. While average privacy improves rapidly with more data (slope $\beta_S \approx -0.5$), the most vulnerable "outlier" samples see their risk decrease much more slowly (slope $\beta_S \approx -0.27$).
Extrapolating this power law reveals a massive "DP Gap." To achieve a level of protection equivalent to a strong DP guarantee ($\epsilon=1$), an average sample might only require a few hundred examples per class. In contrast, protecting the most vulnerable individual samples to the same standard would require millions of samples per class—a scale far beyond most practical fine-tuning scenarios.
Conclusion and Limitations
This work provides a quantitative framework for understanding membership privacy in transfer learning. While the power law offers a tool for risk assessment, it also highlights the difficulty of protecting outliers without formal DP.
Current limitations include the focus on "Head" fine-tuning; training from scratch or full-model fine-tuning may lead to higher memorization. Additionally, the model assumes relatively well-behaved data distributions. Future research will explore how data imbalance and adversary side-knowledge might alter these fundamental privacy laws.
Summary updated