Robin Hood and Matthew Effects - Differential Privacy Has Disparate Impact on Synthetic Data

Georgi Ganev, Bristena Oprisanu, Emiliano De Cristofaro

Abstract

Differential Privacy (DP) is the gold standard for protecting privacy in synthetic data, but it comes with a hidden cost: it reshapes data distributions in ways that can disadvantage minority groups. This paper reveals that DP generative models produce either a "Robin Hood" effect (evening out class sizes) or a "Matthew" effect (amplifying imbalances), both of which lead to disproportionate accuracy drops for underrepresented populations in downstream tasks.

Introduction

As organizations move toward sharing synthetic data to protect individual privacy, they increasingly rely on Differential Privacy (DP) to provide mathematical guarantees against data leakage. While the degradation of performance in DP-trained discriminative models (like classifiers) is well-documented, a more fundamental question remains: What happens when the generative process itself is differentially private?

If a DP model is used to create a synthetic version of a dataset, does it preserve the delicate balance between majority and minority groups? This study investigates whether synthetic data biased by DP leads to unfair or unreliable outcomes for specific subgroups in downstream applications, such as medical diagnostics or credit scoring.

Methodology

The researchers conducted an empirical study using three state-of-the-art DP generative models, each representing a different underlying mechanism:

  1. PrivBayes: Utilizes Bayesian networks and the Laplace Mechanism to approximate data distributions.
  2. DP-WGAN: A Generative Adversarial Network (GAN) that uses DP-SGD (Stochastic Gradient Descent) to sanitize gradients during training.
  3. PATE-GAN: A GAN employing the Private Aggregation of Teacher Ensembles (PATE) framework, where a "student" generator learns from a collection of "teacher" discriminators.

These models were evaluated across four scenarios (S1-S4) involving diverse datasets: Adult (census data), Texas (hospital records), Purchases (customer loyalty), and MNIST (handwritten digits). The researchers measured the size of generated subgroups and the accuracy of classifiers trained on the resulting synthetic data.

The Robin Hood and Matthew Effects

The study identifies two primary ways DP generative models distort data distributions:

  • The Robin Hood Effect: In models like PrivBayes, the addition of Laplace noise followed by "clipping" (to ensure non-negative counts) tends to inflate the size of minority classes while slightly deflating the majority. It effectively "steals" representation from the rich to give to the poor, smoothing out the distribution.
  • The Matthew Effect: Conversely, PATE-GAN often exhibits a "rich get richer" trend. Because the student generator relies on noisy votes from teachers who rarely see minority samples, it often fails to learn rare patterns entirely, further shrinking minority groups in the synthetic output.
Figure 4: Distribution distortion and utility impact across PrivBayes, DP-WGAN, and PATE-GAN. The top row shows how synthetic class sizes deviate from real data (Robin Hood vs. Matthew effects), while the bottom row illustrates the resulting drop in recall relative to class size.

Key Insights

Disparate Accuracy Impact

While utility drops for all groups when using DP synthetic data, the "poor" (minority groups) consistently suffer much larger and more volatile accuracy drops. As the privacy budget ($\epsilon$) decreases (stronger privacy), the accuracy of models trained on synthetic data drops significantly faster for minority subgroups than for the majority. For example, on the Adult dataset, accuracy for the "Female" subgroup dropped much more sharply than for the "Male" subgroup as privacy was tightened.

Artificial Correlations and Failure to Launch

At high privacy levels (low epsilon), PATE-GAN was found to create "ghost" correlations—strong relationships between subgroups and targets that did not exist in the real data. In extreme cases of imbalance (e.g., when a class is only 10% of the size of the majority), some models completely failed to generate any samples of the minority class, rendering the synthetic data useless for those groups.

Figure 3: Baseline performance (no-DP) on the MNIST dataset, showing class size and recall. Note the significant drop for the minority class 8, which is often further exacerbated by DP mechanisms.

Conclusion and Future Work

The study demonstrates that DP generative models rarely preserve original proportions, leading to significant fairness concerns. Future research should focus on developing "fairness-aware" DP training algorithms to prevent these distribution distortions. Additionally, investigating the impact of DP on fairness metrics beyond simple accuracy parity is essential for ensuring that synthetic data remains a viable and equitable tool for privacy-preserving data sharing.

Summary updated

BibTeX
@inproceedings{ganev2022robin,
  author = {Georgi Ganev and Bristena Oprisanu and Emiliano De Cristofaro},
  title = {{Robin Hood and Matthew Effects - Differential Privacy Has Disparate Impact on Synthetic Data}},
  booktitle = {International Conference on Machine Learning},
  year = {2022},
  url = {https://openreview.net/forum?id=xzqGX6pnU5_},
}
Figure