Abstract
Differential Privacy (DP) is the gold standard for protecting privacy in synthetic data, but it comes with a hidden cost: it reshapes data distributions in ways that can disadvantage minority groups. This paper reveals that DP generative models produce either a "Robin Hood" effect (evening out class sizes) or a "Matthew" effect (amplifying imbalances), both of which lead to disproportionate accuracy drops for underrepresented populations in downstream tasks.
Introduction
As organizations move toward sharing synthetic data to protect individual privacy, they increasingly rely on Differential Privacy (DP) to provide mathematical guarantees against data leakage. While the degradation of performance in DP-trained discriminative models (like classifiers) is well-documented, a more fundamental question remains: What happens when the generative process itself is differentially private?
If a DP model is used to create a synthetic version of a dataset, does it preserve the delicate balance between majority and minority groups? This study investigates whether synthetic data biased by DP leads to unfair or unreliable outcomes for specific subgroups in downstream applications, such as medical diagnostics or credit scoring.
Methodology
The researchers conducted an empirical study using three state-of-the-art DP generative models, each representing a different underlying mechanism:
- PrivBayes: Utilizes Bayesian networks and the Laplace Mechanism to approximate data distributions.
- DP-WGAN: A Generative Adversarial Network (GAN) that uses DP-SGD (Stochastic Gradient Descent) to sanitize gradients during training.
- PATE-GAN: A GAN employing the Private Aggregation of Teacher Ensembles (PATE) framework, where a "student" generator learns from a collection of "teacher" discriminators.
These models were evaluated across four scenarios (S1-S4) involving diverse datasets: Adult (census data), Texas (hospital records), Purchases (customer loyalty), and MNIST (handwritten digits). The researchers measured the size of generated subgroups and the accuracy of classifiers trained on the resulting synthetic data.
The Robin Hood and Matthew Effects
The study identifies two primary ways DP generative models distort data distributions:
- The Robin Hood Effect: In models like PrivBayes, the addition of Laplace noise followed by "clipping" (to ensure non-negative counts) tends to inflate the size of minority classes while slightly deflating the majority. It effectively "steals" representation from the rich to give to the poor, smoothing out the distribution.
- The Matthew Effect: Conversely, PATE-GAN often exhibits a "rich get richer" trend. Because the student generator relies on noisy votes from teachers who rarely see minority samples, it often fails to learn rare patterns entirely, further shrinking minority groups in the synthetic output.
Key Insights
Disparate Accuracy Impact
While utility drops for all groups when using DP synthetic data, the "poor" (minority groups) consistently suffer much larger and more volatile accuracy drops. As the privacy budget ($\epsilon$) decreases (stronger privacy), the accuracy of models trained on synthetic data drops significantly faster for minority subgroups than for the majority. For example, on the Adult dataset, accuracy for the "Female" subgroup dropped much more sharply than for the "Male" subgroup as privacy was tightened.
Artificial Correlations and Failure to Launch
At high privacy levels (low epsilon), PATE-GAN was found to create "ghost" correlations—strong relationships between subgroups and targets that did not exist in the real data. In extreme cases of imbalance (e.g., when a class is only 10% of the size of the majority), some models completely failed to generate any samples of the minority class, rendering the synthetic data useless for those groups.
Conclusion and Future Work
The study demonstrates that DP generative models rarely preserve original proportions, leading to significant fairness concerns. Future research should focus on developing "fairness-aware" DP training algorithms to prevent these distribution distortions. Additionally, investigating the impact of DP on fairness metrics beyond simple accuracy parity is essential for ensuring that synthetic data remains a viable and equitable tool for privacy-preserving data sharing.
Summary updated