Abstract
Markov random fields (MRFs) are difficult to evaluate as generative models because computing the test log-probabilities requires the intractable partition function. Annealed importance sampling (AIS) is widely used to estimate MRF partition functions, and often yields quite accurate results. However, AIS is prone to overestimate the log-likelihood with little indication that anything is wrong. We present the Reverse AIS Estimator (RAISE), a stochastic lower bound on the log-likelihood of an approximation to the original MRF model. RAISE requires only the same MCMC transition operators as standard AIS. Experimental results indicate that RAISE agrees closely with AIS log-probability estimates for RBMs, DBMs, and DBNs, but typically errs on the side of underestimating, rather than overestimating, the log-likelihood.
Introduction
In generative modeling, a primary goal is to determine the probability a model assigns to unseen test data. For Markov Random Fields (MRFs), such as Restricted Boltzmann Machines (RBMs) and Deep Boltzmann Machines (DBMs), this calculation is complicated by the partition function ($Z$), a normalizing constant that is computationally intractable for large models.
The current standard, Annealed Importance Sampling (AIS), provides a consistent estimator for $Z$. However, AIS has a known optimistic bias: it tends to underestimate the partition function. Since $Z$ appears in the denominator of the likelihood, underestimating it leads to an overestimation of the log-likelihood. This can lead researchers to report inflated performance metrics without a reliable way to detect the error.
The RAISE Method
To address this, we introduce the Reverse AIS Estimator (RAISE). While standard AIS starts with a simple, tractable distribution and gradually "anneals" it into the complex target distribution, RAISE reverses this process. It begins with an exact sample from the target distribution (conditioned on a test data point) and "melts" it back down to a simple distribution.
RAISE views the AIS process itself as a generative model, known as the "annealing model." By applying Jensen’s Inequality, the expected log of the RAISE weights provides a stochastic lower bound on the log-likelihood of this annealing model. For models with intractable posteriors like DBMs, RAISE incorporates an additional "heating" step to maintain feasibility. To improve computational efficiency, variance reduction via control variates is employed, allowing for per-example evaluation.
Key Insights and Results
The most significant advantage of RAISE is the "Sandwich Strategy." By calculating both AIS (an upper bound) and RAISE (a lower bound), researchers can bracket the true log-likelihood. Convergence is indicated when the two estimates agree closely.
Experiments on MNIST and Omniglot datasets across RBM, DBM, and DBN architectures demonstrate that RAISE consistently provides conservative estimates. A critical finding is the importance of the initial distribution ($p_{ini}$). Using a uniform distribution often results in large gaps between AIS and RAISE. However, employing a "Data Base Rate" (DBR) distribution—which matches the average pixel intensities of the training data—significantly narrows this gap.
On large-scale models with 500+ hidden units, RAISE and AIS estimates converged to within 0.1 nats when using 100,000 intermediate distributions and DBR initialization. RAISE also proved useful in identifying model pathologies; for instance, it can reveal when a model has collapsed into a single state by providing a higher estimate than the model's true likelihood, indicating the annealing model is a better representation than the original MRF.
Limitations and Conclusion
While RAISE offers a more conservative and reliable estimate than AIS, it comes with higher computational costs as it must be run for every test example. Furthermore, RAISE is technically a lower bound on the annealing model approximation, not the original MRF. While these converge as the number of intermediate steps increases, discrepancies can exist for finite steps. Despite these limitations, the combination of AIS and RAISE provides a rigorous framework for evaluating generative models, ensuring that reported performance is both accurate and conservative.
Summary updated