Evaluating Generative Models with Intractable Likelihoods

79 concepts72 papers2 proposals

In classical probabilistic modeling, evaluation is straightforward: you compute the log-likelihood of a held-out test dataset under your model. If the model assigns a high probability density $p(x)$ to real-world data, it has successfully captured the underlying data distribution.

However, modern deep generative models have broken this evaluation paradigm. Many of the most successful architectures—such as Generative Adversarial Networks (GANs), score-based diffusion models, and energy-based frameworks—are implicit generative models. They do not provide an explicit, mathematically tractable probability density function. Instead, they offer a recipe to sample from the distribution by passing random noise through a complex neural network.

Without a tractable likelihood, how can we rigorously evaluate, compare, and diagnose these models? This question has driven a major research direction focused on likelihood-free evaluation and implicit density assessment.

The Likelihood Blindspot and the Failure of Proxies

To understand why evaluating implicit models is difficult, we must first look at why early workarounds failed. When Ian Goodfellow introduced Generative Adversarial Nets, the lack of a tractable likelihood forced researchers to rely on Parzen window density estimation as a proxy. In this approach, a Gaussian distribution is placed over each generated sample to interpolate a continuous density function. However, as noted by Brendan J. Frey in the context of PixelGAN Autoencoders, Parzen window estimates are notoriously unreliable in high-dimensional spaces due to their extreme variance, making them poor tools for ranking models.

This tension culminated in what is known as the "likelihood gap." Aditya Grover et al. exposed this phenomenon in Flow-GAN: Combining Maximum Likelihood and Adversarial Learning in Generative Models by designing an invertible normalizing flow generator that could both generate samples and compute exact log-likelihoods.

The authors discovered a striking divergence: models trained purely to optimize visual sample quality achieved mathematically abysmal likelihood scores, while models optimized for log-likelihood produced poor-looking images. This proved that human visual judgment and statistical goodness-of-fit can completely decouple, rendering qualitative assessment insufficient for true statistical validation.

Disentangling Fidelity and Diversity

When we cannot compute $p(x)$, we are forced to evaluate a model by comparing a finite set of generated samples against a finite set of real data points. The challenge is that a model can fail in two entirely different ways:

  1. Poor Fidelity (Low Quality): The model produces unrealistic samples that look nothing like the real data.
  2. Poor Diversity (Mode Collapse): The model produces highly realistic samples, but only from a tiny subset of the true distribution (e.g., generating only one perfect image of a cat over and over, while ignoring dogs, cars, and houses).

Scalar metrics like the Fréchet Inception Distance (FID) collapse these two failure modes into a single number, making it impossible to diagnose why a model is underperforming.

To solve this, Mehdi S. M. Sajjadi et al. proposed the Precision and Recall for Distributions (PRD) framework in Assessing Generative Models via Precision and Recall.

True Distribution (P)         Generated Distribution (Q)
     [ Mode A ]                      [ Mode A ]  <-- High Precision (Realistic)
     [ Mode B ]                      
         ^
         |--- Missing Mode! <-- Low Recall (Poor Diversity)

The intuition behind PRD mirrors classical information retrieval:

  • Precision measures the fraction of generated samples that fall within the support of the true data distribution. If a model generates unrealistic artifacts, precision drops.
  • Recall measures the fraction of the true data distribution covered by the model. If a model suffers from mode collapse and misses entire categories of data, recall drops.

By framing evaluation as a two-dimensional problem, the authors provided a diagnostic tool that allows researchers to determine whether an implicit model needs better sample quality or a broader coverage of the data space.

Non-Parametric Distance and Relative Similarity

Another way to bypass explicit likelihoods is to use kernel-based methods that measure the distance between two probability distributions using only their samples. The primary tool for this is Maximum Mean Discrepancy (MMD).

In A Test of Relative Similarity for Model Selection in Generative Models Eugene Belilovsky, Eugene Belilovsky et al. introduced a non-parametric statistical hypothesis test to answer a fundamental question: Given two implicit models, which one is significantly closer to the target data distribution? By using MMD as a metric of closeness, the authors built a framework for rigorous model selection without ever needing to evaluate a density function.

To make these sample-based comparisons practical, the statistical power of the test must be maximized. Hsiao-Yu Tung addressed this in Generative Models and Model Criticism via Optimized Maximum Mean Discrepancy by developing a method to automatically optimize the kernel parameters of the MMD test. By tuning the kernel to be as sensitive as possible to the differences between the real and generated distributions, this approach allows practitioners to perform "model criticism"—identifying the exact regions in high-dimensional space where an implicit model fails to match reality.

Furthermore, Yao-Hung Hubert Tsai et al. combined an efficient version of MMD with Post Selection Inference in Selecting the Best in GANs Family: a Post Selection Inference Framework, allowing researchers to confidently select the best model from a large family of candidates while controlling the false positive rate of their statistical conclusions.

Bypassing Densities with Generalized Empirical Likelihoods

Can we achieve the diagnostic granularity of a true likelihood function without actually computing an explicit density? Shakir Mohamed et al. laid the theoretical groundwork for this in Learning in Implicit Generative Models by framing adversarial training as a form of "likelihood-free" inference, where the discriminator acts as a density ratio estimator between the real and generated distributions.

Building on this perspective, Suman V. Ravuri et al. introduced the Generalized Empirical Likelihood (GEL) framework in Understanding Deep Generative Models with Generalized Empirical Likelihoods. GEL provides a powerful, interpretative diagnostic tool by asking a clever question: How much do we need to reweight the samples generated by our model so that their statistical moments perfectly match the moments of the true data?

Generated Samples        GEL Reweighting        True Data Moments
  [ Sample 1 ]  --->      Weight: 0.1      \
  [ Sample 2 ]  --->      Weight: 1.8       +--->   [ Perfect Match ]
  [ Sample 3 ]  --->      Weight: 0.2      /

If a model has captured the true distribution perfectly, every sample receives an equal weight. If the model has dropped a mode, the few samples it does produce near that mode must be heavily upweighted (assigned a high weight) to compensate. Conversely, unrealistic samples are downweighted.

By inspecting these empirical weights, GEL provides both an aggregate performance score and per-sample interpretability, allowing researchers to pinpoint exactly which data features or labels an implicit model is failing to condition on properly.

Domain-Specific Validation and Bounds

The necessity of evaluating implicit models spans far beyond computer vision. In fields like cosmology, Íñigo Zubeldía et al. demonstrated in Extracting cosmological information from the abundance of galaxy clusters with simulation-based inference that Simulation-Based Inference (SBI) can successfully validate complex generative pipelines. By comparing real satellite observations to a distribution of synthetic datasets generated from estimated posteriors, they achieved rigorous statistical validation without requiring an explicit likelihood function.

Meanwhile, for latent variable models where the true marginal likelihood is intractable but bounded, researchers continue to tighten mathematical approximations. In Bounding Evidence and Estimating Log-Likelihood in VAE, Łukasz Struski et al. combined importance sampling with Jensen's inequality to derive a novel upper bound on the variational gap. This allows practitioners to "sandwich" the true, uncomputable log-likelihood between a lower bound (ELBO) and an upper bound, providing a precise window of model performance.

Ultimately, whether through precision-recall decomposition, optimized non-parametric kernel tests, or empirical reweighting, the field has shifted from trying to force implicit models into classical likelihood frameworks to building a new toolkit designed specifically for a likelihood-free world.

Key Foundational Work

The Recurrent Temporal Restricted Boltzmann Machine (2008) — contributed 2 concept elements to this direction.
Discussion of "Spectral Dimensionality Reduction via Maximum Entropy" (2011) — contributed 2 concept elements to this direction.
On the quantitative analysis of deep belief networks (2008) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_likelihood_intractability_8cf367f6,
  title = {likelihood intractability},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/likelihood-intractability/rdr_8cf367f64ecd495095e5c681923ed16a},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_likelihood_intractability_8cf367f6_v1,
  title = {likelihood intractability},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/likelihood-intractability/rdr_8cf367f64ecd495095e5c681923ed16a/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake