Understanding Deep Generative Models with Generalized Empirical Likelihoods

Suman V. Ravuri, Mélanie Rey, Shakir Mohamed, Marc Peter Deisenroth

Abstract

Understanding how well a deep generative model captures a distribution of high-dimensional data remains an important open challenge. It is especially difficult for certain model classes, such as Generative Adversarial Networks and Diffusion Models, whose models do not admit exact likelihoods. In this work, we demonstrate that generalized empirical likelihood (GEL) methods offer a family of diagnostic tools that can identify many deficiencies of deep generative models (DGMs). We show, with appropriate specification of moment conditions, that the proposed method can identify which modes have been dropped, the degree to which DGMs are mode imbalanced, and whether DGMs sufficiently capture intra-class diversity. We show how to combine techniques from Maximum Mean Discrepancy and Generalized Empirical Likelihood to create not only distribution tests that retain per-sample interpretability, but also metrics that include label information. We find that such tests predict the degree of mode dropping and mode imbalance up to 60% better than metrics such as improved precision/recall.

Introduction

As deep generative models (DGMs) reach maturity, producing high-resolution images and realistic audio, the research community is shifting from asking "Is this model good?" to "How exactly is this model failing?" Standard metrics like Fréchet Inception Distance (FID) provide a single aggregate score but offer no insight into which parts of the data distribution are missing. More nuanced metrics, such as Precision and Recall (PR), attempt to quantify "fidelity" and "diversity," but they rely on manifold estimation (typically using k-nearest neighbors), which is highly sensitive to hyperparameters and often biased.

This research addresses the need for a theoretically grounded, hyperparameter-robust, and interpretable evaluation framework that can pinpoint specific deficiencies like mode dropping (missing entire categories), mode imbalance (over-sampling certain types of data), and improper conditioning (ignoring input labels).

Methodology: Generalized Empirical Likelihood

The core of this work is the application of Generalized Empirical Likelihood (GEL) to generative modeling. Instead of comparing entire distributions directly, the authors define "moment conditions"—mathematical rules that the model's output should satisfy if it matches the true data.

Kernel GEL (KGEL)

By combining GEL with Mean Embeddings (ME) and using kernels (like the exponential kernel), images are mapped into a high-dimensional feature space. This allows for distribution tests that are linear in the weights, making the optimization problem computationally tractable.

Two-Sample KGEL (KGEL2) and Label Kernels

To handle "misspecified" models—those generating samples completely outside the support of real data—a two-sample version assigns weights to both real data points and model samples. This identifies which real points the model cannot represent and which model samples are "hallucinations." Furthermore, label kernels extend the framework to conditional models, incorporating hierarchical relationships (e.g., ImageNet) to detect content-label mismatches.

Key Insights and Interpretability

A primary advantage of the GEL framework is per-sample interpretability. Unlike FID, GEL assigns a probability weight to every image in the test set. If a specific real image has a weight of zero, it indicates the model is incapable of representing that specific data type.

Per-sample interpretability using GEL weights to identify model blind spots.

The framework revealed distinct "blind spots": BigGAN-deep struggles with images where a single object fills the frame, whereas Cascaded Diffusion Models (CDM) struggle with "swarms" or groups of objects. Additionally, KGEL is hyperparameter-robust, avoiding the fluctuations seen in Precision/Recall metrics due to 'k' selection in k-NN.

Experimental Results

The approach was validated using CIFAR-10 (for controlled mode-dropping tests) and ImageNet (256x256) to evaluate high-end models like BigGAN-deep, CDM, VQ-VAE2, and ADM.

Comparison of GEL against standard metrics in detecting missing modes across different feature extractors (Pool3 and BYOL).

Key findings include:

  • Superior Detection: KGEL predicts the degree of mode dropping and mode imbalance up to 60% better than "Improved Precision/Recall" metrics.
  • Label Sensitivity: In tests with corrupted labels (30-60% error), KGEL significantly outperformed existing density and coverage metrics, which often failed to notice the mismatch between image content and label.
  • Efficiency: While theoretical complexity is $O(nd^3)$, using Newton’s method on the dual problem makes it practical for modern research workflows.

Conclusion

While KGEL offers significant diagnostic power, it faces computational scaling challenges in very high-dimensional feature spaces, often requiring dimensionality reduction like PCA. For extremely poor models, the optimization may fail (infinite distance), which identifies a "bad" model but provides no gradient for improvement. Future work includes extending this framework to other modalities like text-to-speech or large language models to unlock similar diagnostic capabilities for non-visual data.

Summary updated

Abstract

Understanding how well a deep generative model captures a distribution of high-dimensional data remains an important open challenge. It is especially difficult for certain model classes, such as Generative Adversarial Networks and Diffusion Models, whose models do not admit exact likelihoods. In this work, we demonstrate that generalized empirical likelihood (GEL) methods offer a family of diagnostic tools that can identify many deficiencies of deep generative models (DGMs). We show, with appropriate specification of moment conditions, that the proposed method can identify which modes have been dropped, the degree to which DGMs are mode imbalanced, and whether DGMs sufficiently capture intra-class diversity. We show how to combine techniques from Maximum Mean Discrepancy and Generalized Empirical Likelihood to create not only distribution tests that retain per-sample interpretability, but also metrics that include label information. We find that such tests predict the degree of mode dropping and mode imbalance up to 60% better than metrics such as improved precision/recall.
BibTeX
@inproceedings{ravuri2023understanding,
  author = {Suman V. Ravuri and Mélanie Rey and Shakir Mohamed and Marc Peter Deisenroth},
  title = {{Understanding Deep Generative Models with Generalized Empirical Likelihoods}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2023},
  url = {https://openreview.net/forum?id=LEm6ZSxvqr},
}
Figure