Abstract
Understanding how well a deep generative model captures a distribution of high-dimensional data remains an important open challenge. It is especially difficult for certain model classes, such as Generative Adversarial Networks and Diffusion Models, whose models do not admit exact likelihoods. In this work, we demonstrate that generalized empirical likelihood (GEL) methods offer a family of diagnostic tools that can identify many deficiencies of deep generative models (DGMs). We show, with appropriate specification of moment conditions, that the proposed method can identify which modes have been dropped, the degree to which DGMs are mode imbalanced, and whether DGMs sufficiently capture intra-class diversity. We show how to combine techniques from Maximum Mean Discrepancy and Generalized Empirical Likelihood to create not only distribution tests that retain per-sample interpretability, but also metrics that include label information. We find that such tests predict the degree of mode dropping and mode imbalance up to 60% better than metrics such as improved precision/recall.
Introduction
As deep generative models (DGMs) reach maturity, producing high-resolution images and realistic audio, the research community is shifting from asking "Is this model good?" to "How exactly is this model failing?" Standard metrics like Fréchet Inception Distance (FID) provide a single aggregate score but offer no insight into which parts of the data distribution are missing. More nuanced metrics, such as Precision and Recall (PR), attempt to quantify "fidelity" and "diversity," but they rely on manifold estimation (typically using k-nearest neighbors), which is highly sensitive to hyperparameters and often biased.
This research addresses the need for a theoretically grounded, hyperparameter-robust, and interpretable evaluation framework that can pinpoint specific deficiencies like mode dropping (missing entire categories), mode imbalance (over-sampling certain types of data), and improper conditioning (ignoring input labels).
Methodology: Generalized Empirical Likelihood
The core of this work is the application of Generalized Empirical Likelihood (GEL) to generative modeling. Instead of comparing entire distributions directly, the authors define "moment conditions"—mathematical rules that the model's output should satisfy if it matches the true data.
Kernel GEL (KGEL)
By combining GEL with Mean Embeddings (ME) and using kernels (like the exponential kernel), images are mapped into a high-dimensional feature space. This allows for distribution tests that are linear in the weights, making the optimization problem computationally tractable.
Two-Sample KGEL (KGEL2) and Label Kernels
To handle "misspecified" models—those generating samples completely outside the support of real data—a two-sample version assigns weights to both real data points and model samples. This identifies which real points the model cannot represent and which model samples are "hallucinations." Furthermore, label kernels extend the framework to conditional models, incorporating hierarchical relationships (e.g., ImageNet) to detect content-label mismatches.
Key Insights and Interpretability
A primary advantage of the GEL framework is per-sample interpretability. Unlike FID, GEL assigns a probability weight to every image in the test set. If a specific real image has a weight of zero, it indicates the model is incapable of representing that specific data type.
The framework revealed distinct "blind spots": BigGAN-deep struggles with images where a single object fills the frame, whereas Cascaded Diffusion Models (CDM) struggle with "swarms" or groups of objects. Additionally, KGEL is hyperparameter-robust, avoiding the fluctuations seen in Precision/Recall metrics due to 'k' selection in k-NN.
Experimental Results
The approach was validated using CIFAR-10 (for controlled mode-dropping tests) and ImageNet (256x256) to evaluate high-end models like BigGAN-deep, CDM, VQ-VAE2, and ADM.
Key findings include:
- Superior Detection: KGEL predicts the degree of mode dropping and mode imbalance up to 60% better than "Improved Precision/Recall" metrics.
- Label Sensitivity: In tests with corrupted labels (30-60% error), KGEL significantly outperformed existing density and coverage metrics, which often failed to notice the mismatch between image content and label.
- Efficiency: While theoretical complexity is $O(nd^3)$, using Newton’s method on the dual problem makes it practical for modern research workflows.
Conclusion
While KGEL offers significant diagnostic power, it faces computational scaling challenges in very high-dimensional feature spaces, often requiring dimensionality reduction like PCA. For extremely poor models, the optimization may fail (infinite distance), which identifies a "bad" model but provides no gradient for improvement. Future work includes extending this framework to other modalities like text-to-speech or large language models to unlock similar diagnostic capabilities for non-visual data.
Summary updated