How Faithful is your Synthetic Data? Sample-level Metrics for Evaluating and Auditing Generative Models

Ahmed Alaa, Boris van Breugel, Evgeny Saveliev, Mihaela van der Schaar

Evaluating generative models like GANs and VAEs has long been a "black art." We often rely on metrics like the Inception Score or Fréchet Inception Distance (FID), but these are largely tailored to images and offer little insight into why a model is failing. Are the samples unrealistic? Is the model missing certain types of data? Or is it simply "cheating" by memorizing the training set?

In their paper, "How Faithful is your Synthetic Data? Sample-level Metrics for Evaluating and Auditing Generative Models," Ahmed M. Alaa and colleagues introduce a robust, three-dimensional metric designed to move beyond aggregate scores and provide a granular, sample-level diagnosis of generative performance.

The Evaluation Blindspot

Generative modeling has exploded, but our ability to evaluate these models hasn't kept pace. Existing metrics suffer from three main flaws:

  1. Domain Specificity: Metrics like FID are built for images and don't translate well to tabular or time-series data.
  2. Lack of Interpretability: A single number (like a likelihood score) doesn't tell you if a model is suffering from "mode collapse" (lacking diversity) or "mode invention" (generating unrealistic outliers).
  3. The Overfitting Blindspot: Most metrics ignore whether a model is actually "inventing" new data or just copying the training set—a massive problem for privacy-sensitive applications like healthcare.
Visualizing synthetic vs. real data distributions to identify authentic, unauthentic, and noisy samples.

A 3D Framework: Precision, Recall, and Authenticity

To address these issues, the authors propose a new 3D evaluation metric—($\alpha$-Precision, $\beta$-Recall, Authenticity)—that characterizes the fidelity, diversity, and generalization of any generative model. The core of the paper is the move from "hard" supports to "soft" density-based supports using minimum volume sets.

  • $\alpha$-Precision (Fidelity): Measures how many synthetic samples fall within the "typical" regions of the real data distribution. By sweeping $\alpha$ from 0 to 1, researchers can generate a curve that shows how realistic samples are at different levels of "typicality."
  • $\beta$-Recall (Diversity): Measures how much of the real data distribution is covered by the synthetic samples. It detects if the model is missing entire categories of data.
  • Authenticity (Generalization): This is the "secret sauce." It uses a likelihood-ratio test to determine if a synthetic sample is a novel creation or a noisy copy of a training point. If a sample is too close to a training point relative to that point's nearest neighbors, it's flagged as "unauthentic."

Technically, the authors implement this by embedding data into a feature space using one-class neural networks (hyperspheres). This allows the metric to be domain-agnostic; as long as you can embed the data, you can measure it.

Sample-Level Auditing

Unlike previous metrics, this framework works at the individual sample level. Because these metrics can be calculated for each generated sample, you can use them as an active filter rather than just a passive grade.

The proposed evaluation and auditing pipelines, demonstrating how sample-level metrics filter out low-quality generations.

If a GAN produces 1,000 images, you can automatically discard the 100 that have low $\alpha$-Precision or low Authenticity. This effectively "cleans" the model's output after training, improving the overall model performance in a post-hoc manner.

Key Insights and Experimental Results

The authors demonstrate the versatility of their metrics across several distinct data types, including tabular COVID-19 patient records (SIVEP-GRIPE), complex ICU time-series data (AmsterdamUMCdb), and standard image benchmarks (MNIST & CIFAR-10).

  • Predictive Power: In the COVID-19 patient data experiment, the authors' $\text{IP}_\alpha$ (Integrated Precision) metric correctly ranked generative models according to how well a predictive model trained on that synthetic data would perform on real data. Competing metrics like FID and standard Precision failed to provide the correct ranking.
  • Disentangling Failure Modes: The paper shows that standard Precision/Recall metrics are often "blind" to density shifts. A model can have perfect standard precision while still being poor because it over-represents outliers. The $\alpha/\beta$ framework catches this.
  • Post-hoc Improvement: By "auditing" an ADS-GAN model (discarding unauthentic samples), the authors improved the downstream AUC-ROC of a mortality prediction model from 0.76 to 0.78 without retraining the generator.
  • The Privacy-Utility Tradeoff: By using the Authenticity metric, the authors show they can tune hyper-parameters to find the "sweet spot" where a model generates high-quality data without leaking private training information.

Limitations and Future Work

While the framework is powerful, it relies heavily on the quality of the evaluation embedding ($\Phi$). If the embedding space doesn't capture the relevant features of the data, the metrics won't be meaningful. The authors also note that while they provide a non-parametric way to estimate these supports, choosing the right hyper-parameters (like the number of neighbors $k$) still requires some care.

Future work could involve integrating these metrics directly into the loss functions of generative models, potentially forcing models to be "authentic" and "precise" by design rather than relying solely on post-hoc auditing.

Summary updated

Abstract

Devising domain- and model-agnostic evaluation metrics for generative models is an important and as yet unresolved problem. Most existing metrics, which were tailored solely to the image synthesis setup, exhibit a limited capacity for diagnosing the modes of failure of generative models across broader application domains. In this paper, we introduce a 3-dimensional evaluation metric, ($\alpha$-Precision, $\beta$-Recall, Authenticity), that characterizes the fidelity, diversity and generalization performance of any generative model in a domain-agnostic fashion. Our metric unifies statistical divergence measures with precision-recall analysis, enabling sample-level and distribution-level diagnoses of model fidelity and diversity. We introduce generalization as an additional dimension for model performance that quantifies the extent to which a model copies training data---a crucial performance indicator when modeling sensitive data with requirements on privacy. The three metric components correspond to (interpretable) probabilistic quantities, and are estimated via sample-level binary classification. The sample-level nature of our metric inspires a novel use case which we call model auditing, wherein we judge the quality of individual samples generated by a (black-box) model, discarding low-quality samples and hence improving the overall model performance in a post-hoc manner.
BibTeX
@inproceedings{alaa2022how,
  author = {Ahmed Alaa and Boris van Breugel and Evgeny Saveliev and Mihaela van der Schaar},
  title = {{How Faithful is your Synthetic Data? Sample-level Metrics for Evaluating and Auditing Generative Models}},
  booktitle = {International Conference on Learning Representations},
  year = {2022},
  url = {https://openreview.net/forum?id=8qWazUd8Jm},
}
Figure