Differential Privacy Has Disparate Impact on Model Accuracy

Eugene Bagdasaryan, Vitaly Shmatikov

Abstract: Differential privacy (DP) is a popular mechanism for training machine learning models with bounded leakage about the presence of specific points in the training data. The cost of differential privacy is a reduction in the model's accuracy. We demonstrate that in the neural networks trained using differentially private stochastic gradient descent (DP-SGD), this cost is not borne equally: accuracy of DP models drops much more for the underrepresented classes and subgroups. For example, a gender classification model trained using DP-SGD exhibits much lower accuracy for black faces than for white faces. Critically, this gap is bigger in the DP model than in the non-DP model, i.e., if the original model is unfair, the unfairness becomes worse once DP is applied. We demonstrate this effect for a variety of tasks and models, including sentiment analysis of text and image classification. We then explain why DP training mechanisms such as gradient clipping and noise addition have disproportionate effect on the underrepresented and more complex subgroups, resulting in a disparate reduction of model accuracy.

Introduction

Differential Privacy (DP) is widely regarded as the gold standard for protecting individual data in machine learning. However, research from Cornell Tech reveals a significant trade-off: the cost of privacy is not distributed equally. While DP-SGD (Differentially Private Stochastic Gradient Descent) is known to reduce overall model accuracy, this study demonstrates that it disproportionately harms underrepresented subgroups and complex classes, effectively amplifying existing biases.

The Mechanics of Disparate Impact

The researchers identify two primary mechanisms within DP-SGD that drive this inequality:

  1. Gradient Clipping: To bound the influence of any single data point, gradients are clipped to a maximum norm ($S$). Underrepresented or "complex" points, which the model hasn't yet mastered, typically produce larger gradients. Clipping these gradients reduces the effective learning rate for these specific groups more than for the majority.
  2. Noise Addition: Gaussian noise is added to the clipped gradients. Because the signal from minority groups is already weakened by clipping and their smaller sample size, the noise easily overwhelms their contribution to the model updates.

Experimental Evidence

The study evaluated these effects across diverse domains, including facial recognition, sentiment analysis, and species classification.

Figure 3: Disparate impact on facial recognition and general trends across subgroups. (a) shows accuracy gaps widening between skin tones under DP. (b) and (c) illustrate how smaller subgroup sizes and lower initial accuracy correlate with larger performance drops.

In facial recognition tasks using the Diversity in Faces dataset, the accuracy gap between lighter-skinned and darker-skinned individuals widened significantly as privacy constraints became stricter. Similarly, in sentiment analysis, accuracy for African-American English (AAE) plummeted under DP compared to Standard American English (SAE).

Figure 5: Accuracy vs. Class Label for the iNaturalist dataset. The DP model's performance drops precipitously for underrepresented species (right side of the x-axis) compared to the non-DP baseline.

For species classification on the iNaturalist dataset, which follows a long-tail distribution, the DP model's performance on rare species was often no better than random guessing, even when the non-DP model performed reasonably well.

Key Insights: "The Poor Become Poorer"

The most consistent finding is that subgroups with lower initial accuracy in a standard model suffer the most significant losses when DP is introduced. This "bias amplification" means that DP doesn't just preserve unfairness; it exacerbates it. Furthermore, complexity acts as a target: in language models, users with more diverse vocabularies saw larger accuracy drops than those with simpler language patterns.

Conclusion and Future Work

The findings suggest that privacy and fairness cannot be treated as independent goals. Current fairness interventions, like oversampling, are often mathematically incompatible with DP requirements. Future research must focus on developing "private and fair" algorithms that do not force a trade-off where the most vulnerable populations lose the most utility. Simply increasing the privacy budget (\epsilon) is not a viable solution, as it undermines the core purpose of using differential privacy.

Summary updated

Abstract

Differential privacy (DP) is a popular mechanism for training machine learning models with bounded leakage about the presence of specific points in the training data. The cost of differential privacy is a reduction in the model's accuracy. We demonstrate that in the neural networks trained using differentially private stochastic gradient descent (DP-SGD), this cost is not borne equally: accuracy of DP models drops much more for the underrepresented classes and subgroups. For example, a gender classification model trained using DP-SGD exhibits much lower accuracy for black faces than for white faces. Critically, this gap is bigger in the DP model than in the non-DP model, i.e., if the original model is unfair, the unfairness becomes worse once DP is applied. We demonstrate this effect for a variety of tasks and models, including sentiment analysis of text and image classification. We then explain why DP training mechanisms such as gradient clipping and noise addition have disproportionate effect on the underrepresented and more complex subgroups, resulting in a disparate reduction of model accuracy.
BibTeX
@inproceedings{bagdasaryan2019differential,
  author = {Eugene Bagdasaryan and Vitaly Shmatikov},
  title = {{Differential Privacy Has Disparate Impact on Model Accuracy}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2019},
  url = {https://openreview.net/forum?id=FnK70ujOEq},
}
Figure