Abstract
There is significant growth and interest in the use of synthetic data as an enabler for machine learning in environments where the release of real data is restricted due to privacy or availability constraints. Despite a large number of methods for synthetic data generation, there are comparatively few results on the statistical properties of models learnt on synthetic data, and fewer still for situations where a researcher wishes to augment real data with another party’s synthesised data. We use a Bayesian paradigm to characterise the updating of model parameters when learning in these settings, demonstrating that caution should be taken when applying conventional learning algorithms without appropriate consideration of the synthetic data generating process and learning task at hand. Recent results from general Bayesian updating support a novel and robust approach to Bayesian synthetic-learning founded on decision theory that outperforms standard approaches across repeated experiments on supervised learning and inference problems.
Introduction
To protect user privacy, organizations increasingly release synthetic data—artificially generated datasets that maintain the statistical properties of the original data while satisfying Differential Privacy (DP) guarantees. However, to guarantee privacy, the algorithms that generate synthetic data must inject noise or alter distributions. This means the Synthetic Data Generating Process (S-DGP) is fundamentally different from the true Data Generating Process (DGP).
Standard machine learning and Bayesian inference assume the data you train on comes from the true DGP. If a standard model is trained on an infinite amount of synthetic data, it will perfectly learn the synthetic distribution, which is misspecified by design. Consequently, a practitioner trying to augment their small real-world dataset with synthetic data might actually degrade their model's performance by pulling it away from the true underlying reality.
Methodology
The authors frame learning from synthetic data as a problem of Bayesian inference under model misspecification. Instead of using standard Bayesian updating, they utilize General Bayesian Updating (Minimum Divergence Inference).
Robust Loss Functions
Standard Bayesian updating relies on the log-likelihood, which minimizes the Kullback-Leibler (KL) divergence. Because DP synthetic data often contains heavy-tailed noise or "outliers," KL divergence performs poorly. The authors replace it with robust scoring rules:
- Re-weighted log-likelihood: Scaling the impact of synthetic data by a weight $w < 1$.
- $\beta$-divergence ($\beta$D): A divergence measure naturally robust to outliers.
Learning Trajectories and Optimal Stopping
The authors map out a "learning trajectory" to track how model performance changes as more synthetic data is added. They propose calculating an optimal number of synthetic samples ($m^*$) to use, stopping the learning process before the model is pulled too far toward the S-DGP.
Predictive Averaging
To leverage synthetic data beyond the optimal stopping point $m^*$, they demonstrate that one can train multiple models on chunks of size $m^*$ and average their posterior predictive distributions without performance degradation.
Experimental Setup
The methods were evaluated on both simulated and real-world datasets:
- Simulated Gaussian: A univariate Gaussian dataset where the S-DGP is created using the Laplace mechanism for Differential Privacy.
- UCI Heart Disease & Framingham Cohort: Real-world medical datasets used for binary classification (Bayesian Logistic Regression). Synthetic versions were generated using PATE-GAN.
Key Insights and Results
A crucial observation is that more synthetic data is not always better. Performance typically improves initially as synthetic data is added, hits a peak, and then degrades as the model overfits to the privacy-preserving noise of the S-DGP.
Robustness vs. Naivety
The $\beta$-divergence updating method consistently outperforms standard Bayesian updating. By being skeptical of S-DGP outliers, $\beta$D safely extracts the useful signal from the synthetic data while ignoring the privacy-induced noise. In contrast, naive models (treating synthetic data as real) often performed worse than models using no synthetic data at all, especially when a small baseline of real data was already available.
The GAN "Sweet Spot"
Counter-intuitively, for GAN-generated data, a stricter privacy guarantee (lower $\epsilon$) sometimes yielded better downstream model performance for small sample sizes. This is hypothesized to be due to mode collapse in the GAN, where injected noise acts as a regularizer representing the true DGP well for small samples.
Conclusion
The primary limitation is that calculating the optimal stopping point ($m^*$) requires access to a holdout set of real data. Future work suggests moving beyond general robust divergences toward tailored adjustments that explicitly model the specific mathematical quirks of the chosen synthetic data generation mechanism.
Summary updated