Abstract
Despite the tremendous progress in the estimation of generative models, the development of tools for diagnosing their failures and assessing their performance has advanced at a much slower pace. Recent research has proposed separating evaluation into precision and recall, but these methods often rely on problematic data quantization. This paper introduces a unified framework for Precision-Recall (P-R) curves based on information divergence frontiers, providing a principled approach for both discrete and continuous settings.
Introduction
Evaluating deep generative models, such as GANs and VAEs, is notoriously difficult. Standard metrics like the Fréchet Inception Distance (FID) collapse complex performance characteristics into a single scalar, failing to distinguish between low sample quality (precision) and poor distribution coverage (recall). Consequently, if a model receives a poor score, it is unclear whether the failure stems from generating low-quality samples or from missing entire modes of the target distribution.
Previous attempts to create P-R curves for distributions often relied on data quantization—grouping data into discrete bins or clusters. This is problematic because results are highly sensitive to the choice of clustering and do not translate well to continuous distributions. This paper proposes a unified theory that works for both discrete and continuous cases without requiring arbitrary discretization.
Methodology: Information Divergence Frontiers
The core innovation of this work is the introduction of an auxiliary distribution, $R$, that acts as a bridge between the target distribution $P$ and the model distribution $Q$.
Instead of a single metric, the authors analyze the trade-off between two divergences: the distance from $P$ to $R$ and the distance from $Q$ to $R$. By varying $R$ to minimize these distances simultaneously, they trace a "Pareto frontier" which effectively forms a Precision-Recall curve. The framework utilizes the family of Rényi divergences ($D_\alpha$), where the parameter $\alpha$ allows practitioners to control the metric's sensitivity to outliers or missing modes.
Key Insights: The Geometry of Evaluation
The framework distinguishes between two types of frontiers to capture different aspects of distribution overlap:
- Exclusive Frontiers: Focus on the intersection of $P$ and $Q$ ($R_\cap$), used to define precision and recall by penalizing over-coverage and under-coverage.
- Inclusive Frontiers: Focus on the union of $P$ and $Q$ ($R_\cup$), providing a broader perspective on how the distributions overlap.
As illustrated above, different discrepancies between the target and model distributions result in distinct P-R curve profiles. This geometric approach allows for a clear diagnosis of model failures. The authors further demonstrate that computing these curves is equivalent to "walking" along a geodesic path between two distributions in a specific information-geometric space.
Results and Discussion
The paper successfully bridges the gap between information theory (divergences) and information retrieval (precision/recall). A key finding is that popular existing metrics, such as the PRD metric by Sajjadi et al. (2018), are actually special cases of this framework where the Rényi parameter $\alpha$ goes to infinity.
Crucially, the authors provide a formal analysis of how discretization affects evaluation, showing that coarse quantization introduces a systematic positive bias that can make a model appear more performant than it truly is. By deriving analytical solutions for exponential families (like Gaussians), the framework remains computationally efficient for many standard generative modeling tasks.
Conclusion
While this framework provides a robust theoretical foundation for continuous P-R curves, challenges remain in high-dimensional density estimation for raw pixel space. Future work will explore extending these algorithms to richer model families and investigating how different divergence choices might better capture the perceptual qualities of generated data.
Summary updated