How Faithful is your Synthetic Data? Sample-level Metrics for Evaluating and Auditing Generative Models

Ahmed M. Alaa, Boris van Breugel, Evgeny Saveliev, Mihaela van der Schaar

Abstract

Evaluating generative models is notoriously tricky, often relying on domain-specific or distribution-level metrics that obscure specific modes of failure. This paper introduces a domain-agnostic, 3-dimensional evaluation framework—($\alpha$-Precision, $\beta$-Recall, Authenticity)—that assesses fidelity, diversity, and generalization at the sample level. By evaluating individual samples rather than just the overall distribution, this approach enables "model auditing": automatically filtering out low-quality or memorized synthetic data to improve a model's output post-hoc.

Introduction

As generative models (GANs, VAEs, Diffusion models) become ubiquitous, evaluating their performance remains a massive challenge. Traditional likelihood scores scale poorly in high dimensions, while popular alternatives like Fréchet Inception Distance (FID) are heavily tailored to images and evaluate the generated distribution as a whole.

Standard Precision and Recall metrics have critical blind spots: they only check if the supports (boundaries) of the real and synthetic distributions overlap, ignoring the actual density. A model could generate 99% of its samples in a tiny corner of the real data distribution and still score perfectly on standard precision. Furthermore, existing metrics often ignore generalization—a model can achieve perfect fidelity and diversity simply by memorizing training data, a catastrophic failure for privacy-sensitive applications.

Conceptual framework of sample-level metrics: $\alpha$-Precision (fidelity), $\beta$-Recall (diversity), and Authenticity (generalization).

Methodology

The proposed framework captures three key dimensions of generative performance:

  • $\alpha$-Precision (Fidelity): Measures the fraction of synthetic samples that resemble the "most typical" fraction $\alpha$ of real samples, penalizing unrealistic "noisy" samples.
  • $\beta$-Recall (Diversity): Measures the fraction of real samples covered by the most typical fraction $\beta$ of synthetic samples, identifying "mode collapse."
  • Authenticity (Generalization): Quantifies the rate at which a model invents new samples rather than copying training data. It uses a hypothesis test based on the distance between a synthetic sample and its nearest neighbor in the training set.

To compute these metrics, the authors use a One-Class Neural Network (Deep SVDD) to embed data into a hypersphere, separating "typical" samples near the center from "outliers" near the boundary. $k$-Nearest Neighbors ($k$-NN) in this embedded space is then used to estimate the supports. Crucially, these per-sample metrics enable Model Auditing, allowing practitioners to reject individual samples that are unrealistic or memorized.

Experimental Results

The authors validated their metrics across tabular (COVID-19 records), time-series (ICU data), and image (MNIST, CIFAR-10) datasets.

Model Ranking and Downstream Performance

In predictive modeling tasks, the integrated $\alpha$-Precision ($IP_\alpha$) perfectly ranked generative models according to their actual downstream AUC. In contrast, standard metrics like FID often ranked poorer models higher. For example, on the COVID-19 tabular dataset, $IP_\alpha$ and $IR_\beta$ provided a more reliable ground-truth ranking than traditional benchmarks.

The Power of Auditing

Auditing provides a "free lunch" for improving synthetic data quality. By filtering out unauthentic or imprecise samples post-hoc, the researchers significantly improved the downstream predictive AUC of an ADS-GAN model from 0.76 to 0.78.

Results showing (a) model ranking accuracy, (b) hyper-parameter sensitivity, and (c) the improvement in precision and authenticity through post-hoc auditing.

Identifying Privacy Risks

The Authenticity metric revealed critical flaws in existing models. In a NeurIPS privacy challenge, the "winning" model was found to have terrible Authenticity—it was essentially adding slight noise to real patient data, a major privacy risk that standard metrics failed to detect.

Conclusion

By moving from distribution-level to sample-level evaluation, this framework provides a more nuanced and robust assessment of generative models. The ability to audit models post-hoc offers a practical path to improving the fidelity and privacy of synthetic datasets without the need for expensive retraining. Future work will explore applying these metrics to evaluate fairness, specifically checking if models systematically drop minority subgroups.

Summary updated

Abstract

Devising domain- and model-agnostic evaluation metrics for generative models is an important and as yet unresolved problem. Most existing metrics, which were tailored solely to the image synthesis setup, exhibit a limited capacity for diagnosing the different modes of failure of generative models across broader application domains. In this paper, we introduce a 3-dimensional evaluation metric, ($\alpha$-Precision, $\beta$-Recall, Authenticity), that characterizes the fidelity, diversity and generalization performance of any generative model in a domain-agnostic fashion. Our metric unifies statistical divergence measures with precision-recall analysis, enabling sample- and distribution-level diagnoses of model fidelity and diversity. We introduce generalization as an additional, independent dimension (to the fidelity-diversity trade-off) that quantifies the extent to which a model copies training data -- a crucial performance indicator when modeling sensitive data with requirements on privacy. The three metric components correspond to (interpretable) probabilistic quantities, and are estimated via sample-level binary classification. The sample-level nature of our metric inspires a novel use case which we call model auditing, wherein we judge the quality of individual samples generated by a (black-box) model, discarding low-quality samples and hence improving the overall model performance in a post-hoc manner.
BibTeX
@inproceedings{alaa2022how,
  author = {Ahmed M. Alaa and Boris van Breugel and Evgeny Saveliev and Mihaela van der Schaar},
  title = {{How Faithful is your Synthetic Data? Sample-level Metrics for Evaluating and Auditing Generative Models}},
  booktitle = {International Conference on Learning Representations},
  year = {2022},
  url = {https://openreview.net/forum?id=8qWazUd8Jm},
}
Figure