Differential Privacy Has Disparate Impact on Model Accuracy

Eugene Bagdasaryan, Omid Poursaeed, Vitaly Shmatikov

While Differential Privacy (DP) is widely considered the gold standard for protecting training data, it comes with a hidden social cost: it disproportionately reduces the accuracy of models for underrepresented and minority subgroups. In essence, the standard training mechanism for DP acts as a "regressive tax" on accuracy, where the groups already underserved by machine learning models pay the highest price for privacy.

The Hidden Cost of Privacy

Differential Privacy is designed to ensure that a machine learning model doesn't "leak" specific information about individuals in its training set. The standard method for achieving this in deep learning is DP-SGD (Differentially Private Stochastic Gradient Descent). DP-SGD modifies standard training through two core mechanisms:

  1. Gradient Clipping: Limiting the influence of any single data point by capping its gradient norm at a specific threshold.
  2. Noise Addition: Adding Gaussian noise to the aggregated gradients to mask individual contributions.

While researchers have long known that DP causes a slight drop in overall model accuracy, this paper investigates a critical question: Is this accuracy loss shared equally? The authors explore whether the "privacy tax" falls harder on minority groups or complex data samples, and whether applying DP to an already biased model makes that bias better, worse, or keeps it the same.

Why DP-SGD Penalizes Minorities

The study reveals a striking "poor become poorer" dynamic: subgroups that already have lower accuracy in a standard (non-DP) model suffer the largest accuracy drops when DP is introduced. DP doesn't just preserve unfairness; it exacerbates it. This happens for two primary reasons:

  • The Gradient Conflict: Underrepresented or "complex" data points often produce gradients with larger norms because they are "surprising" to the model. DP-SGD clips these large gradients more aggressively than the smaller gradients of the majority. This effectively reduces the learning rate for minority groups, preventing the model from ever learning their features properly.
  • Noise vs. Signal: Because minority groups have fewer samples, their total gradient contribution is small. When random noise is added, it easily drowns out the signal from these small groups, while the majority signal remains strong enough to survive the noise.

Key Findings: Bias Amplification

To demonstrate that this is a general phenomenon rather than an artifact of one specific task, the authors evaluated models trained with DP-SGD across several domains, including facial recognition, natural language processing, and image classification.

In a gender classification task using facial images, the gap between accuracy for lighter-skinned and darker-skinned faces grew significantly wider under DP. For example, a non-DP model might perform reasonably well for both groups, but under DP, the accuracy for the darker-skinned group plummets disproportionately.

Accuracy drop for darker skin tones under DP

Results Across Domains

This disparate impact extends well beyond facial recognition. The authors observed similar trends in both text sentiment analysis and complex image classification tasks.

  • Sentiment Analysis: When classifying the sentiment of tweets, the accuracy for users writing in African-American English (AAE) dropped significantly more than for those using Standard American English (SAE) once DP was applied.
  • Species Classification: On the highly imbalanced iNaturalist dataset, common classes like Aves (birds) saw almost no change under DP. However, underrepresented classes like Actinopterygii (ray-finned fishes) saw their accuracy drop from roughly 35% to near zero.
Disparate impact on sentiment analysis and species classification

In controlled experiments using the MNIST dataset (where the digit "8" was artificially made a minority), the non-DP model achieved 97% accuracy on "8". Under DP, accuracy for the majority class ("2") stayed at 98%, but accuracy for the minority ("8") dropped to 77%. While increasing the batch size can mitigate some of the noise impact, it often requires a higher "privacy budget" ($\epsilon$), creating a difficult trade-off between fairness, privacy, and utility.

Limitations and Future Work

The authors acknowledge that they have identified a fundamental tension in privacy-preserving machine learning, but a perfect solution remains elusive.

Critically, common techniques used to fix dataset imbalance—such as oversampling or re-weighting—are mathematically incompatible with the strict sensitivity bounds required by DP-SGD. We cannot simply "bolt on" privacy to existing models and expect them to remain fair. Future work must focus on developing novel training mechanisms that provide robust differential privacy guarantees without disproportionately silencing the "long tail" of the data distribution.

Summary updated

Abstract

Differential privacy (DP) is a popular mechanism for training machine learning models with bounded leakage about the presence of specific points in the training data. The cost of differential privacy is a reduction in the model's accuracy. We demonstrate that in the neural networks trained using differentially private stochastic gradient descent (DP-SGD), this cost is not borne equally: accuracy of DP models drops much more for the underrepresented classes and subgroups. For example, a gender classification model trained using DP-SGD exhibits much lower accuracy for black faces than for white faces. Critically, this gap is bigger in the DP model than in the non-DP model, i.e., if the original model is unfair, the unfairness becomes worse once DP is applied. We demonstrate this effect for a variety of tasks and models, including sentiment analysis of text and image classification. We then explain why DP training mechanisms such as gradient clipping and noise addition have disproportionate effect on the underrepresented and more complex subgroups, resulting in a disparate reduction of model accuracy.
BibTeX
@inproceedings{bagdasaryan2019differential,
  author = {Eugene Bagdasaryan and Omid Poursaeed and Vitaly Shmatikov},
  title = {{Differential Privacy Has Disparate Impact on Model Accuracy}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2019},
  url = {https://openreview.net/forum?id=FnK70ujOEq},
}
Figure