The Neural Autoregressive Distribution Estimator

Hugo Larochelle, Iain MurrayAISTATS 2011 · OpenReview

Introduction

Modeling the joint distribution of high-dimensional discrete data is a fundamental challenge in generative modeling. Traditionally, researchers have relied on Restricted Boltzmann Machines (RBMs), which offer significant representational power but suffer from a major drawback: they are undirected models with intractable partition functions. Calculating the exact probability of a data point in an RBM requires exponential summation, necessitating expensive approximations like Gibbs sampling for both training and evaluation.

Alternatively, Fully Visible Bayesian Networks (FVBNs) provide a tractable approach by decomposing the joint probability into a product of conditionals using the chain rule: $p(x_1, ..., x_D) = \prod p(x_i | x_{<i})$. However, standard FVBNs often lack the modeling capacity of RBMs. The Neural Autoregressive Distribution Estimator (NADE) bridges this gap by parameterizing these conditional distributions with a neural network architecture that shares weights across the sequence, achieving RBM-like power with the tractability of a directed model.

Methods and Architecture

NADE computes the probability of each variable $v_i$ conditioned on all preceding variables in a fixed ordering. For each step $i$, it computes a hidden state $\mathbf{h}_i$ which is then used to output the conditional probability $p(v_i = 1 | \mathbf{v}_{<i})$.

The Weight-Sharing Trick

A naive implementation of a neural FVBN would require $O(HD^2)$ complexity, where $H$ is the hidden layer size and $D$ is the input dimensionality, because a new hidden layer would be computed from scratch for every variable. NADE introduces a clever weight-sharing mechanism: by using the same weight matrix $\mathbf{W}$ for all conditional distributions, the hidden state $\mathbf{h}_{i+1}$ can be updated incrementally from $\mathbf{h}_i$ by simply adding the contribution of the new observation $v_i$. This reduces the total complexity to $O(HD)$, making it as efficient as a standard RBM or a single pass through an autoencoder.

Comparison of the FVSBN and NADE architectures. NADE uses a hidden layer structure inspired by RBMs while maintaining the directed, autoregressive structure of an FVBN.

Key Results

The authors evaluated NADE across eight diverse datasets, including UCI benchmarks (Adult, DNA, Mushrooms), text data (RCV1, NIPS), and binarized MNIST images. Performance was measured using average test log-likelihood.

As shown in the table below, NADE consistently outperformed other tractable models, such as Mixtures of Bernoullis (MoB) and standard FVSBNs, on 6 out of the 8 datasets. On the MNIST dataset, NADE achieved a test log-likelihood of -88.86, which is remarkably close to the performance of large RBMs (approx. -86.34) despite NADE's exact tractability. Unlike RBMs, NADE allows for exact, one-pass sampling and direct calculation of the probability of any given observation.

Average test log-likelihoods (relative to MoB) across eight datasets. NADE demonstrates superior performance compared to other tractable alternatives in most cases.

Discussion and Future Work

NADE represents a significant step forward in tractable generative modeling. It can be viewed as a "truncated" version of an RBM that performs a single iteration of mean-field inference, or as a specifically constrained autoencoder.

While highly effective, NADE is sensitive to the ordering of variables, particularly in structured data like images. Future research directions include extending NADE to continuous data using different output layers (e.g., Gaussian) and increasing the model's depth. The "Deep NADE" variant, which stacks multiple hidden layers, is expected to further enhance the model's ability to capture complex dependencies in high-dimensional data.

Summary updated

BibTeX
@inproceedings{larochelle2011neural,
  author = {Hugo Larochelle and Iain Murray},
  title = {{The Neural Autoregressive Distribution Estimator}},
  booktitle = {International Conference on Artificial Intelligence and Statistics},
  year = {2011},
  url = {https://openreview.net/forum?id=UNwiW5D6h_x},
}
Figure