Abstract
Differential privacy is a strong notion for privacy that can be used to prove formal guarantees, in terms of a privacy budget, $\epsilon$, about how much information is leaked by a mechanism. However, implementations of privacy-preserving machine learning often select large values of $\epsilon$ in order to get acceptable utility of the model, with little understanding of the impact of such choices on meaningful privacy. Moreover, in scenarios where iterative learning procedures are used, differential privacy variants that offer tighter analyses are used which appear to reduce the needed privacy budget but present poorly understood trade-offs between privacy and utility. In this paper, we quantify the impact of these choices on privacy in experiments with logistic regression and neural network models. Our main finding is that there is a huge gap between the upper bounds on privacy loss that can be guaranteed, even with advanced mechanisms, and the effective privacy loss that can be measured using current inference attacks. Current mechanisms for differentially private machine learning rarely offer acceptable utility-privacy trade-offs with guarantees for complex learning tasks: settings that provide limited accuracy loss provide meaningless privacy guarantees, and settings that provide strong privacy guarantees result in useless models.
Introduction
Differential Privacy (DP) has emerged as the gold standard for protecting individual data in datasets. By utilizing a "privacy budget" ($\epsilon$), it bounds the influence any single record can have on a model's output. Theoretically, a small $\epsilon$ (e.g., $\le 1.0$) signifies robust privacy. However, training complex models like neural networks involves iterative processes where privacy loss accumulates. To maintain a low total $\epsilon$, researchers often add significant noise, which frequently compromises model utility. Consequently, the community has turned to "relaxed" DP variants, such as Rényi DP (RDP), which employ sophisticated mathematical accounting to provide tighter privacy bounds. This research investigates whether these mathematical relaxations offer genuine real-world protection or merely serve as accounting mechanisms that obscure empirical risks.
Methodology and DP Variants
The study evaluates several gradient perturbation mechanisms—where noise is added to gradients during training—across various DP frameworks. These include:
- Naive Composition (NC): The most conservative summation of privacy loss.
- Advanced Composition (AC): Provides a tighter theoretical bound than NC.
- Zero-Concentrated DP (zCDP) & Rényi DP (RDP): Modern variants that facilitate higher utility by assuming privacy loss follows a sub-Gaussian distribution.
To evaluate empirical privacy, the researchers deployed membership inference attacks (determining if a specific record was in the training set) and attribute inference attacks (guessing sensitive attributes from partial information).
Experimental Setup
The evaluation focused on two complex multi-class classification benchmarks:
- CIFAR-100: 60,000 images across 100 classes.
- Purchase-100: 200,000 customer purchase records grouped into 100 styles.
These datasets were selected because they highlight the stark trade-offs between privacy and accuracy in complex learning tasks.
Key Insights: The Theory-Practice Gap
A primary finding is the "Meaningless Guarantee." To achieve usable accuracy on complex datasets, the required $\epsilon$ often reaches values of 100 or even 1000. Mathematically, an $\epsilon$ of 1000 allows the probability of a data leak to increase by a factor of $e^{1000}$, offering no practical protection.
Furthermore, while variants like RDP and zCDP improve utility by adding less noise for a given $\epsilon$ budget, this directly increases the success rate of membership inference attacks compared to more conservative DP methods. The study also emphasizes that "per-instance clipping" is essential; without it, models fail to learn effectively under DP constraints.
Results and Empirical Leakage
- Accuracy vs. Privacy: On CIFAR-100, models with strong privacy ($\epsilon \le 10$) achieved only random-guess levels of accuracy (~1%). Modest utility required $\epsilon > 100$.
- Attack Success: In high-utility private models (RDP at $\epsilon = 1000$), membership inference attacks identified thousands of training members with a Positive Predictive Value (PPV) up to 0.82.
- Baseline Observations: Non-private models sometimes exhibited lower-than-expected leakage to simple attacks, suggesting that some observed "privacy" in DP models may stem from architectural properties rather than the DP mechanism itself.
Conclusion
The research reveals a significant "gulf" between theoretical privacy upper bounds ($e^\epsilon - 1$) and measurable empirical leakage. While current models may be safer than the loose mathematical bounds suggest, the guarantees themselves are often too weak to be meaningful in practice. The authors conclude that the field requires a fundamental shift: either developing new mechanisms for better utility at low $\epsilon$ or re-evaluating how privacy risks are communicated when high $\epsilon$ values are used.
Code for these experiments is available at: https://github.com/bargavj/EvaluatingDPML
Summary updated