Introduction
Evaluating generative models is a notoriously difficult task. Ideally, we want to measure the "log-likelihood"—the probability the model assigns to real-world test data. However, many modern architectures, such as Restricted Boltzmann Machines (RBMs) or Deep Belief Networks (DBNs), involve intractable normalization constants that make exact calculation impossible.
Even more challenging, models like Denoising Auto-encoders or Generative Stochastic Networks (GSNs) lack an explicit probability formula entirely; they only provide a procedure to generate samples. Previous attempts to evaluate these models relied on Parzen density estimators applied to generated samples, which require tedious tuning of bandwidth parameters. Other methods, like Annealed Importance Sampling (AIS), often yield "optimistic" estimates that can mislead researchers into overestimating a model's true performance.
To address this, researchers have introduced the Conservative Sampling-based Likelihood (CSL) estimator. CSL is a robust method for evaluating generative models that lack a tractable likelihood function. Unlike previous methods, CSL provides a mathematically proven lower bound on test log-likelihood and naturally accounts for the "mixing" quality of the model's sampling procedure.
The CSL Estimator
The core idea behind the CSL estimator is to leverage the latent variables ($h$) within the model. Instead of solely examining the final generated samples ($x$), CSL evaluates the conditional probability $P(x|h)$.
The Algorithm: 1. Run the model's Markov chain to collect a set of latent variable samples $S$. 2. For a given test data point $x$, calculate the average of $P(x|h)$ across all sampled $h$ values. 3. The logarithm of this average serves as the estimate.
Advantages of CSL: * No Tuning Required: It eliminates the need for a manually tuned bandwidth parameter because the model’s own conditional distribution acts as a natural "kernel." * Lower Variance: By utilizing the conditional distribution $P(x|h)$ rather than raw samples, the estimator effectively summarizes a vast number of potential outcomes, leading to significantly more stable results. * Conservative by Design: Using Jensen’s inequality, the authors prove that this estimator forms a lower bound on the true log-likelihood. If a model's Markov chain mixes poorly (e.g., it gets stuck in one region of the data space), the CSL score will be low. This naturally penalizes models that are difficult to sample from.
The authors also introduce a Biased CSL variant. By initializing the Markov chain directly at the test sample $x$, the estimator can explore the "neighborhood" of the data point much faster. This makes it highly useful for comparing models even when computational constraints limit the number of samples.
Key Insights
- Mixing as a Metric: A profound insight from this work is that an evaluation metric should reflect how well a model mixes. If a model learns the data distribution perfectly but its sampling chain is broken, it is not practically useful. CSL effectively captures this failure mode.
- Rao-Blackwellization: The transition from using raw samples to using the conditional expectation $P(x|h)$ is a form of Rao-Blackwellization, a classic statistical technique for reducing the variance of an estimator.
- The "Pessimism" Advantage: In scientific evaluation, a conservative estimate (a lower bound) is generally preferable to an optimistic one. CSL ensures that high scores are genuinely earned.
Experimental Results
The researchers evaluated their method using the binarized MNIST dataset, a standard benchmark in generative modeling.
Convergence: On a small RBM where the "true" likelihood could be calculated exactly, the CSL estimator converged perfectly to the ground truth as the number of samples increased.
Model Comparison: The "Biased CSL" variant demonstrated an extremely high correlation with both the true log-likelihood and AIS estimates across different RBM configurations. This proves it can be used to reliably rank models by quality, tracking the true probability much more faithfully than AIS, which can diverge.
Scalability: The authors successfully applied CSL to complex architectures like Generative Stochastic Networks (GSNs) and Deep Boltzmann Machines (DBMs), providing a reliable way to monitor training progress where previous methods fell short.
Limitations and Future Work
While powerful, the CSL approach has a few limitations:
- Computational Cost: The unbiased version of CSL requires a large number of samples from the Markov chain to converge, particularly if the model exhibits poor mixing properties.
- Requirement of $P(x|h)$: The method strictly requires that the model has a computable conditional distribution. While common in many latent variable models, it may not apply to every generative architecture (such as certain types of GANs).
- Bias in the "Biased" Version: While excellent for ranking models quickly, the biased variant tends to over-estimate the likelihood, meaning it loses the strict "conservative lower bound" guarantee of the original version.
Future work will likely involve a more systematic study of how CSL behaves across an even wider variety of generative architectures, further solidifying its role as a standard evaluation metric.
Summary updated