Introduction
Evaluating generative models like Generative Adversarial Networks (GANs) and Variational Autoencoders (VAEs) remains a significant challenge in machine learning. For years, the Fréchet Inception Distance (FID) has served as the primary benchmark. However, FID is a composite metric that conflates two distinct properties: fidelity (the realism of individual samples) and diversity (the extent to which the model captures the full variety of the target distribution).
To disentangle these qualities, researchers previously introduced Precision and Recall (P&R). Precision was intended to measure fidelity, while Recall targeted diversity. Despite their conceptual appeal, recent analysis reveals that existing P&R implementations are "not ready for practical use" due to extreme sensitivity to outliers, failure to pass the "identity test" (where a dataset compared to itself should yield a perfect score), and a lack of principled hyperparameter selection.
Density and Coverage: A New Framework
This paper introduces Density and Coverage (D&C), two metrics designed to overcome the structural flaws of P&R. Both metrics utilize $k$-nearest neighbor (kNN) manifolds but redefine how these manifolds are constructed and queried.
Density (Fidelity)
Traditional Precision checks if a generated sample falls within the union of neighborhood spheres around real samples. This binary approach is vulnerable to outliers; a single real outlier can create a large "valid" region that misrepresents the true distribution. Density improves upon this by counting how many neighborhood spheres of real samples contain a given generated sample. This "superposition" approach rewards samples that land in high-density regions of the real data, providing a more robust measure of fidelity.
Coverage (Diversity)
Recall typically builds a manifold around the fake samples to see how much of the real data it covers. Because generated samples are often noisy, this fake manifold can expand erratically, leading to inflated Recall scores. Coverage reverses this logic by anchoring the manifold to the real samples. It measures the fraction of real-sample neighborhoods that contain at least one generated sample. This ensures that the metric is grounded in the stable, ground-truth distribution.
Theoretical Foundation and Hyperparameters
A core contribution of this work is the analytic derivation of D&C. The authors mathematically determine the expected values of these metrics when the real and fake distributions are identical. This theoretical grounding allows researchers to move away from arbitrary hyperparameter choices.
By analyzing the convergence of these metrics, the authors demonstrate that $k=5$ and a sample size of $N=10,000$ are sufficient to achieve stable, reliable results with a 95% confidence level. This standardization is crucial for making results comparable across different studies.
Experimental Validation
The authors validated D&C across diverse datasets, including FFHQ (faces), MNIST (digits), and audio spectrograms. Two key findings highlight the superiority of D&C over P&R:
- Robustness to Outliers: In experiments on the CelebA dataset, the introduction of outliers caused traditional Recall to fluctuate by up to 15%. In contrast, Coverage remained stable, showing less than a 2% change.
- Detection of Mode Dropping: When a model fails to generate certain categories (e.g., missing specific digits in MNIST), Coverage exhibits a smooth, linear decline. Traditional Recall often remains deceptively high until almost all diversity is lost, failing to provide an early warning of model failure.
The Embedding Bias
Most evaluation metrics rely on feature extractors (like Inception-v3) pre-trained on ImageNet. The authors show that these embeddings are biased toward natural images. For "out-of-domain" data like audio or medical imagery, pre-trained features may not be optimal. Interestingly, the paper suggests that randomly initialized CNNs can serve as effective, unbiased feature extractors for macroscopic evaluation in these specialized domains.
Conclusion
Density and Coverage provide a more reliable and interpretable alternative to Precision and Recall. By anchoring diversity measures to the real data manifold and rewarding fidelity through density superposition, these metrics offer a clearer picture of generative model performance. While the choice of embedding remains a factor, the analytic framework and robustness of D&C mark a significant step toward standardized evaluation in generative modeling.
Summary updated