Foundations of Bayesian Learning from Synthetic Data

Harrison Wilde, Jack Jewson, Sebastian J. Vollmer, Chris C. Holmes

Introduction

Synthetic data is frequently proposed as a solution for machine learning in privacy-constrained sectors such as healthcare and finance. By generating artificial datasets that mimic real-world distributions, researchers aim to leverage large-scale data without compromising individual privacy. However, recent research titled "Foundations of Bayesian Learning from Synthetic Data" reveals a fundamental flaw in this approach: treating synthetic data as if it were real can lead to model degradation. As more synthetic data is introduced, models may paradoxically become less accurate by converging toward the biases of the generator rather than the true underlying distribution.

The Synthetic Data Trap

The core issue lies in the "Synthetic Data Generating Process" (S-DGP). Whether using Differential Privacy (DP) to inject noise or generative models like GANs that may suffer from mode collapse, synthetic data is inherently a misspecified representation of reality.

Standard Bayesian updating assumes that data is drawn from the "true" distribution. When fed synthetic data, a naive Bayesian model learns the synthetic distribution with increasing precision. Unfortunately, because the synthetic distribution is biased, the model eventually prioritizes learning the generator's errors over the actual phenomenon. This leads to a "turning point" where additional synthetic samples actually harm performance.

A Generalized Bayesian Framework

To address these challenges, the authors propose a robust framework based on Generalized Bayesian learning.

1. Robust Loss Functions ($eta$-Divergence)

Instead of standard log-likelihood, which minimizes Kullback-Leibler (KL) divergence, the researchers employ $eta$-divergence ($eta$D). This loss function is inherently "skeptical," adaptively downweighting observations that appear as outliers—a common occurrence in DP-noised datasets. By using $eta$D, the model remains focused on the more reliable portions of the synthetic data.

2. Learning Trajectories and Optimal Stopping

The researchers analyze "learning trajectories"—the path of model performance as synthetic data is added. They demonstrate that performance often follows a U-shaped curve (for error) or an inverted U-shape (for accuracy metrics like AUROC). By using a small validation set of real data, practitioners can identify the optimal number of synthetic samples ($m^*$) to include before the "turning point" is reached.

Figure 2: AUROC performance on the UCI Heart Dataset (real n=30). Note the inverted U-shape indicating a "turning point" where additional synthetic samples begin to degrade performance, and the superior robustness of β-divergence configurations.

Key Results and Insights

The framework was validated using simulated Gaussian models, the UCI Heart Disease dataset, and the larger Framingham Cohort dataset.

  • Effective Sample Gain: One of the most significant findings is the quantification of synthetic data's value. In the Framingham dataset experiments, using the optimal amount of synthetic data provided a performance boost equivalent to gaining 50 to 75 additional real samples. This is particularly valuable in small-data regimes where every real sample is costly or difficult to obtain.
Figure 5: Maximum effective real sample gain through the use of synthetic data across different real sample sizes. The gain is most pronounced in smaller real-data regimes.
  • The Privacy Paradox: Interestingly, higher privacy noise (lower $\epsilon$) can sometimes act as a regularizer. In some cases, it prevents a generative model from overfitting to specific real-world quirks, making the resulting synthetic data safer for a robust learner to use in limited quantities.
  • Predictive Averaging: The authors prove that averaging predictions across multiple synthetic realizations is a "free lunch," consistently improving performance over training on a single, larger synthetic dataset.

Conclusion

The "Foundations of Bayesian Learning from Synthetic Data" provides a critical theoretical and practical roadmap for using artificial data safely. By recognizing the inherent misspecification in synthetic generators and employing robust methods like $\beta$-divergence, researchers can extract genuine value from synthetic data while avoiding the trap of learning from noise. Future work aims to extend these foundations to complex deep learning architectures and develop "noise-aware" models that explicitly account for specific privacy mechanisms.

Summary updated

BibTeX
@inproceedings{wilde2021foundations,
  author = {Harrison Wilde and Jack Jewson and Sebastian J. Vollmer and Chris C. Holmes},
  title = {{Foundations of Bayesian Learning from Synthetic Data}},
  booktitle = {International Conference on Artificial Intelligence and Statistics},
  year = {2021},
  url = {https://openreview.net/forum?id=fTra2eXuEU},
}
Figure