Abstract
Evaluating generative models is notoriously tricky, often relying on domain-specific or distribution-level metrics that obscure specific modes of failure. This paper introduces a domain-agnostic, 3-dimensional evaluation framework—($\alpha$-Precision, $\beta$-Recall, Authenticity)—that assesses fidelity, diversity, and generalization at the sample level. By evaluating individual samples rather than just the overall distribution, this approach enables "model auditing": automatically filtering out low-quality or memorized synthetic data to improve a model's output post-hoc.
Introduction
As generative models (GANs, VAEs, Diffusion models) become ubiquitous, evaluating their performance remains a massive challenge. Traditional likelihood scores scale poorly in high dimensions, while popular alternatives like Fréchet Inception Distance (FID) are heavily tailored to images and evaluate the generated distribution as a whole.
Standard Precision and Recall metrics have critical blind spots: they only check if the supports (boundaries) of the real and synthetic distributions overlap, ignoring the actual density. A model could generate 99% of its samples in a tiny corner of the real data distribution and still score perfectly on standard precision. Furthermore, existing metrics often ignore generalization—a model can achieve perfect fidelity and diversity simply by memorizing training data, a catastrophic failure for privacy-sensitive applications.
Methodology
The proposed framework captures three key dimensions of generative performance:
- $\alpha$-Precision (Fidelity): Measures the fraction of synthetic samples that resemble the "most typical" fraction $\alpha$ of real samples, penalizing unrealistic "noisy" samples.
- $\beta$-Recall (Diversity): Measures the fraction of real samples covered by the most typical fraction $\beta$ of synthetic samples, identifying "mode collapse."
- Authenticity (Generalization): Quantifies the rate at which a model invents new samples rather than copying training data. It uses a hypothesis test based on the distance between a synthetic sample and its nearest neighbor in the training set.
To compute these metrics, the authors use a One-Class Neural Network (Deep SVDD) to embed data into a hypersphere, separating "typical" samples near the center from "outliers" near the boundary. $k$-Nearest Neighbors ($k$-NN) in this embedded space is then used to estimate the supports. Crucially, these per-sample metrics enable Model Auditing, allowing practitioners to reject individual samples that are unrealistic or memorized.
Experimental Results
The authors validated their metrics across tabular (COVID-19 records), time-series (ICU data), and image (MNIST, CIFAR-10) datasets.
Model Ranking and Downstream Performance
In predictive modeling tasks, the integrated $\alpha$-Precision ($IP_\alpha$) perfectly ranked generative models according to their actual downstream AUC. In contrast, standard metrics like FID often ranked poorer models higher. For example, on the COVID-19 tabular dataset, $IP_\alpha$ and $IR_\beta$ provided a more reliable ground-truth ranking than traditional benchmarks.
The Power of Auditing
Auditing provides a "free lunch" for improving synthetic data quality. By filtering out unauthentic or imprecise samples post-hoc, the researchers significantly improved the downstream predictive AUC of an ADS-GAN model from 0.76 to 0.78.
Identifying Privacy Risks
The Authenticity metric revealed critical flaws in existing models. In a NeurIPS privacy challenge, the "winning" model was found to have terrible Authenticity—it was essentially adding slight noise to real patient data, a major privacy risk that standard metrics failed to detect.
Conclusion
By moving from distribution-level to sample-level evaluation, this framework provides a more nuanced and robust assessment of generative models. The ability to audit models post-hoc offers a practical path to improving the fidelity and privacy of synthetic datasets without the need for expensive retraining. Future work will explore applying these metrics to evaluate fairness, specifically checking if models systematically drop minority subgroups.
Summary updated