An Architecture for Deep, Hierarchical Generative Models

Philip Bachman

Overview

Training deep, directed generative models—such as Variational Autoencoders (VAEs)—becomes notoriously difficult as more layers of latent variables are added. Each stochastic layer introduces variance into gradient estimates, acting as "noise" that drowns out the subtle signals needed to learn sophisticated data structures. While models like DRAW utilize "sequential depth" (iterative refinement over time), they often reuse the same latent variables, making it difficult for the model to learn a true hierarchy of abstractions from simple edges to complex objects.

To address this, researchers introduce Matryoshka Networks (MatNets). Named after Russian nesting dolls due to their deeply nested structure, MatNets feature a hierarchical architecture that enables the stable training of generative models with over 10 layers of latent variables. By combining deterministic "shortcut" paths with a hybrid approach that pairs global hierarchical structure with local autoregressive refinement, MatNets achieve state-of-the-art performance in image modeling and complex tasks like unsupervised character discovery and image inpainting.

The MatNet Architecture

The MatNet architecture is designed to overcome vanishing gradients and training instability in hierarchically deep networks. It consists of three primary components:

  1. Bottom-Up (BU) Network: A deterministic encoder that processes the input to create a hierarchy of features.
  2. Top-Down (TD) Network: The generative component that produces the output through "iterative stochastic refinement," where each layer adds a small amount of detail to the previous layer's state.
  3. Merge Modules: The "glue" of the system. During training, they combine information from the BU and TD networks to produce the approximate posterior distributions for the latent variables.
MatNet Architecture Diagram

Key Innovations

  • Deterministic Paths: The model uses residual and shortcut connections to provide direct paths between all latent variables and the final output. This ensures that even high-level latents have a clear "voice" in the final reconstruction, making them easier to train.
  • Hybrid Modeling: To handle the "pixel-perfect" details of natural images (like sharp edges), the architecture incorporates a lightweight autoregressive model (inspired by PixelCNN) into the final reconstruction step. This offloads the burden of modeling local textures from the latent variables, allowing them to focus on global structure.
  • Mixture-based Priors: Instead of a simple Gaussian prior, the model can use a Gaussian Mixture Model (GMM) for the top-most latent variables to capture multi-modal data distributions.

Key Insights and Results

The model was evaluated on several standard benchmarks, including MNIST, Omniglot, CIFAR-10, and high-resolution (64x64) CelebA and LSUN datasets.

State-of-the-Art Performance

At the time of publication, MatNets achieved leading Negative Log-Likelihood (NLL) scores across multiple datasets. For instance, the convolutional MatNet achieved an NLL of $\le 78.5$ on MNIST and $\le 89.5$ on Omniglot. On CIFAR-10, the MatNet equipped with the autoregressive component achieved a highly competitive score of $< 3.24$ bits-per-pixel.

Negative Log-Likelihood Results

Structural Learning and Imputation

MatNets prove that hierarchical depth is a powerful, complementary tool to sequential refinement. This was strikingly demonstrated during qualitative evaluations of image inpainting on LSUN images. The model successfully performed "inpainting" on 64x64 images with large occlusions. In a notable "watermark" moment, the model learned to reconstruct Shutterstock watermarks during inpainting, proving it wasn't just blurring pixels but actively learning the underlying structural regularities of the dataset.

Furthermore, by using a GMM prior, the MatNet was able to group Omniglot characters by stylistic similarity without any labels, effectively "discovering" latent classes in an unsupervised manner.

Limitations and Future Work

While highly effective, the MatNet architecture is relatively complex. It involves three distinct sub-networks and specific "merge" logic, which may be harder to implement than standard VAEs. Additionally, while the autoregressive component helps with local textures, the authors note that the model's ability to capture global structure provides a relatively weak benefit to the final NLL score compared to simply getting the local edges right.

Future directions suggest combining the hierarchical depth of MatNets with the sequential depth of models like DRAW. There is also significant potential in applying this architecture to non-image data, such as language or sequence modeling, where hierarchical abstractions are equally critical.

Summary updated

Abstract

We present an architecture which lets us train deep, directed generative models with many layers of latent variables. We include deterministic paths between all latent variables and the generated output, and provide a richer set of connections between computations for inference and generation, which enables more effective communication of information throughout the model during training. To improve performance on natural images, we incorporate a lightweight autoregressive model in the reconstruction distribution. These techniques permit end-to-end training of models with 10+ layers of latent variables. Experiments show that our approach achieves state-of-the-art performance on standard image modelling benchmarks, can expose latent class structure in the absence of label information, and can provide convincing imputations of occluded regions in natural images.
BibTeX
@inproceedings{bachman2016architecture,
  author = {Philip Bachman},
  title = {{An Architecture for Deep, Hierarchical Generative Models}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2016},
  url = {https://openreview.net/forum?id=H1ESk_-uWB},
}
Figure