Abstract: Differential privacy (DP) is a popular mechanism for training machine learning models with bounded leakage about the presence of specific points in the training data. The cost of differential privacy is a reduction in the model's accuracy. We demonstrate that in the neural networks trained using differentially private stochastic gradient descent (DP-SGD), this cost is not borne equally: accuracy of DP models drops much more for the underrepresented classes and subgroups. For example, a gender classification model trained using DP-SGD exhibits much lower accuracy for black faces than for white faces. Critically, this gap is bigger in the DP model than in the non-DP model, i.e., if the original model is unfair, the unfairness becomes worse once DP is applied. We demonstrate this effect for a variety of tasks and models, including sentiment analysis of text and image classification. We then explain why DP training mechanisms such as gradient clipping and noise addition have disproportionate effect on the underrepresented and more complex subgroups, resulting in a disparate reduction of model accuracy.
Introduction
Differential Privacy (DP) is widely regarded as the gold standard for protecting individual data in machine learning. However, research from Cornell Tech reveals a significant trade-off: the cost of privacy is not distributed equally. While DP-SGD (Differentially Private Stochastic Gradient Descent) is known to reduce overall model accuracy, this study demonstrates that it disproportionately harms underrepresented subgroups and complex classes, effectively amplifying existing biases.
The Mechanics of Disparate Impact
The researchers identify two primary mechanisms within DP-SGD that drive this inequality:
- Gradient Clipping: To bound the influence of any single data point, gradients are clipped to a maximum norm ($S$). Underrepresented or "complex" points, which the model hasn't yet mastered, typically produce larger gradients. Clipping these gradients reduces the effective learning rate for these specific groups more than for the majority.
- Noise Addition: Gaussian noise is added to the clipped gradients. Because the signal from minority groups is already weakened by clipping and their smaller sample size, the noise easily overwhelms their contribution to the model updates.
Experimental Evidence
The study evaluated these effects across diverse domains, including facial recognition, sentiment analysis, and species classification.
In facial recognition tasks using the Diversity in Faces dataset, the accuracy gap between lighter-skinned and darker-skinned individuals widened significantly as privacy constraints became stricter. Similarly, in sentiment analysis, accuracy for African-American English (AAE) plummeted under DP compared to Standard American English (SAE).
For species classification on the iNaturalist dataset, which follows a long-tail distribution, the DP model's performance on rare species was often no better than random guessing, even when the non-DP model performed reasonably well.
Key Insights: "The Poor Become Poorer"
The most consistent finding is that subgroups with lower initial accuracy in a standard model suffer the most significant losses when DP is introduced. This "bias amplification" means that DP doesn't just preserve unfairness; it exacerbates it. Furthermore, complexity acts as a target: in language models, users with more diverse vocabularies saw larger accuracy drops than those with simpler language patterns.
Conclusion and Future Work
The findings suggest that privacy and fairness cannot be treated as independent goals. Current fairness interventions, like oversampling, are often mathematically incompatible with DP requirements. Future research must focus on developing "private and fair" algorithms that do not force a trade-off where the most vulnerable populations lose the most utility. Simply increasing the privacy budget (\epsilon) is not a viable solution, as it undermines the core purpose of using differential privacy.
Summary updated