Why Does Differential Privacy with Large Epsilon Defend Against Practical Membership Inference Attacks?

Andrew Lowy, Zhuohang Li, Jing Liu, Toshiaki Koike-Akino, Kieran Parsons, Ye WangarXiv 2024 · DOI · arXiv

Introduction

Differential Privacy (DP) has long been the gold standard for privacy-preserving machine learning, providing a mathematical guarantee that the presence or absence of a single individual in a dataset does not significantly alter the output of an algorithm. This guarantee is controlled by the privacy budget, $\epsilon$. Theoretically, for $\epsilon$ to provide meaningful protection against Membership Inference Attacks (MIAs), it should be small (typically $\epsilon < 1$).

However, a significant gap exists between this theory and real-world practice. Tech giants like Apple and Google frequently deploy models with $\epsilon$ values as high as 7 or 14. According to standard DP theory, an attacker should be able to identify training set members with near-certainty ($>99.9\%$) at these levels. Yet, empirically, these models remain remarkably resistant to state-of-the-art attacks. This paper, presented at AAAI 2024, introduces the Practical Membership Privacy (PMP) framework to resolve this paradox.

The Practical Membership Privacy (PMP) Framework

The authors argue that standard DP is "overkill" because it assumes a worst-case adversary: one who knows every record in the dataset except for the target individual. In reality, practical attackers are much less informed.

PMP models a more realistic scenario where the attacker knows the data comes from a "parent set" $X$ (e.g., a specific subpopulation) but is uncertain about which specific $n$ samples from that set were used for training. By accounting for this uncertainty, the authors show that a seemingly weak (large) DP guarantee often translates into a much stronger, meaningful practical safeguard.

Key Insights: Uncertainty as a Shield

The core "aha moment" of the research is that an attacker’s lack of knowledge about the rest of the dataset makes it significantly harder to isolate the influence of a target individual. This creates a "Ratio Gap" where a large DP $\epsilon$ hides a very small PMP $\tilde{\epsilon}$. For instance, a mechanism with $\epsilon = 7.5$ might actually provide the practical protection equivalent to $\tilde{\epsilon} = 0.1$.

The Role of Dimensionality

One of the most striking findings is the impact of data dimensionality. While high-dimensional data is often seen as a privacy risk, the PMP framework reveals that the gap between theoretical DP and practical risk actually grows as dimensionality increases.

Average ratios vs. dimensionality showing the widening gap between theoretical DP and practical PMP as dimensions increase.

As shown in the figure above, the ratio $\tilde{\epsilon}/\epsilon$ decreases as dimensionality grows, suggesting that in high-dimensional spaces, the practical risk of membership inference drops much faster than worst-case theory suggests.

Impact of Outliers and Clipping

In DP-SGD, clipping thresholds are used to bound the influence of individual gradients. Large clipping thresholds allow outliers to "leak" more information to a worst-case attacker, causing the DP $\epsilon$ to explode. However, PMP demonstrates that if the attacker does not know which specific points are outliers, the practical risk remains low.

The effect of clip thresholds on the ratio between PMP and DP parameters.

The experimental results indicate that as the clip threshold increases, the ratio of practical risk to theoretical budget drops significantly. This explains why practitioners can use relatively high clipping thresholds to maintain model utility without sacrificing practical privacy.

Conclusion and Guidance for Practitioners

The PMP framework provides a bridge between the rigorous guarantees of DP and the empirical observations of privacy researchers. By proving that the success probability of any practical MIA is bounded by $1/(1 + e^{-\tilde{\epsilon}})$, the authors offer a way to translate desired practical protection levels into specific DP parameters based on data characteristics.

While PMP is not intended to replace DP—which remains the safer choice for formal, worst-case guarantees—it serves as a vital tool for understanding why "vacuous" DP budgets are effective in the wild. It encourages a more nuanced view of privacy that considers the realistic capabilities of potential adversaries.

Summary updated

Abstract

For small privacy parameter $\epsilon$, $\epsilon$-differential privacy (DP) provides a strong worst-case guarantee that no membership inference attack (MIA) can succeed at determining whether a person's data was used to train a machine learning model. The guarantee of DP is worst-case because: a) it holds even if the attacker already knows the records of all but one person in the data set; and b) it holds uniformly over all data sets. In practical applications, such a worst-case guarantee may be overkill: practical attackers may lack exact knowledge of (nearly all of) the private data, and our data set might be easier to defend, in some sense, than the worst-case data set. Such considerations have motivated the industrial deployment of DP models with large privacy parameter (e.g. $\epsilon \geq 7$), and it has been observed empirically that DP with large $\epsilon$ can successfully defend against state-of-the-art MIAs. Existing DP theory cannot explain these empirical findings: e.g., the theoretical privacy guarantees of $\epsilon \geq 7$ are essentially vacuous. In this paper, we aim to close this gap between theory and practice and understand why a large DP parameter can prevent practical MIAs. To tackle this problem, we propose a new privacy notion called practical membership privacy (PMP). PMP models a practical attacker's uncertainty about the contents of the private data. The PMP parameter has a natural interpretation in terms of the success rate of a practical MIA on a given data set. We quantitatively analyze the PMP parameter of two fundamental DP mechanisms: the exponential mechanism and Gaussian mechanism. Our analysis reveals that a large DP parameter often translates into a much smaller PMP parameter, which guarantees strong privacy against practical MIAs. Using our findings, we offer principled guidance for practitioners in choosing the DP parameter.
BibTeX
@misc{lowy2024why,
  author = {Andrew Lowy and Zhuohang Li and Jing Liu and Toshiaki Koike-Akino and Kieran Parsons and Ye Wang},
  title = {{Why Does Differential Privacy with Large Epsilon Defend Against Practical Membership Inference Attacks?}},
  year = {2024},
  eprint = {2402.09540},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2402.09540},
}
Figure