Factoring Variations in Natural Images with Deep Gaussian Mixture Models

Aäron van den Oord, Benjamin SchrauwenNeurIPS 2014

TL;DR

This paper introduces the Deep Gaussian Mixture Model (Deep GMM), a scalable generative model that stacks multiple layers of GMMs to capture complex variations in natural images. By using a clever parameter-tying scheme, the model achieves the representational power of an exponential number of mixture components while remaining computationally efficient and highly parallelizable.

What Problem Does This Paper Address?

In the world of unsupervised learning, researchers face a trade-off: Deep Learning models offer immense representational power but are often difficult to scale and train without labels. Conversely, Shallow Models like the Gaussian Mixture Model (GMM) are easy to scale and understand but lack the "depth" to capture the intricate, hierarchical variations found in real-world data like images.

Traditional GMMs struggle with "combinatorial explosions." For example, if an image can have 10 different lighting conditions and 10 different rotations, a standard GMM needs 100 components to model every combination. The authors aim to bridge this gap by creating a deep architecture that can factorize these variations (e.g., modeling lighting in one layer and rotation in another), allowing for a much more compact and efficient representation.

Methods

The core innovation is the Deep GMM, which is constructed by stacking multiple GMM layers.

Graphical models comparing a standard Gaussian, a shallow GMM, and the proposed Deep GMM.
  1. Hierarchical Transformations: Instead of picking one Gaussian component from a flat list, the model selects a "path" through several layers of linear transformations. A path consists of one transformation matrix from each layer.
  2. Parameter Tying: While a Deep GMM with $k$ layers and $N$ components per layer is mathematically equivalent to a shallow GMM with $N^k$ components, it only requires training $N \times k$ parameters. This "parameter tying" forces the model to learn shared features across different image types, significantly reducing overfitting.
  3. The EM Algorithm: To train the model, the authors adapted the Expectation-Maximization (EM) algorithm:
  4. E-Step (Heuristic): Since calculating every possible path is computationally impossible (exponential growth), they use a "hard-EM" approach with a coordinate ascent heuristic to quickly find the most likely path for each data point.
  5. M-Step (Folding): They introduce a "folding" technique where all layers above the current one are treated as a single transformation. This allows them to optimize each layer independently using standard tools like L-BFGS-B.
  6. Parallelization: Unlike many deep models that rely on Stochastic Gradient Descent (SGD), the Deep GMM is parallelizable by design. Both the E-step and M-step can be distributed across multiple machines with minimal communication.

Datasets

The authors evaluated their model on two primary benchmarks for density estimation:

  • BSDS300 (Berkeley Segmentation Dataset): They used 8x8 grayscale patches, preprocessed to 63 dimensions (removing the mean pixel value).
  • Tiny Images: A dataset of low-resolution (8x8) images used to test how the model handles more diverse and difficult visual data.

Key Insights and Ideas

  • Factoring Variations: The "aha moment" is that natural images are products of independent variations (e.g., brightness $\times$ contrast $\times$ pose). By stacking layers, the Deep GMM models these as a sequence of transformations, which is much more natural than trying to memorize every possible combination.
  • Depth Prevents Overfitting: The experiments showed that as you add more components to a shallow GMM, it eventually starts to overfit. However, Deep GMMs with the same number of parameters continue to generalize better, proving that the structure of the model acts as a powerful regularizer.
  • Interpretability: Unlike many "black box" neural networks, the transformations in a Deep GMM are explicit linear operations, making the model's internal logic more transparent.

Key Results

  • State-of-the-Art Performance: On the BSDS300 dataset, a 3-layer Deep GMM achieved an average log-likelihood of 156.2.
  • Comparison: This outperformed standard GMMs (153.7), Student-t Mixture Models (155.3), and single-layer RNADE models. It was only surpassed by an ensemble of RNADE models (157.0), which is significantly more computationally expensive.
  • Efficiency: The model showed a clear "plateau" in performance for shallow models that the deep versions were able to break through, achieving better results with fewer total parameters.
Log-likelihood performance on the BSDS300 dataset as the number of components increases.

Limitations and Future Work

  • Patch-Based Constraints: Currently, the model is limited to small image patches (8x8). Scaling this to full-sized high-resolution images remains a challenge.
  • Dimensionality: The use of Kronecker products in the optimization step makes the model computationally expensive if the input dimensionality is very high.
  • Future Directions: The authors suggest incorporating convolutions (locally connected filters) to handle larger images and extending the framework to model discrete data (like text) rather than just continuous pixel values. They also aim to compare the model against other deep density estimators like Restricted Boltzmann Machines (RBMs).

Summary updated

BibTeX
@inproceedings{oord2014factoring,
  author = {Aäron van den Oord and Benjamin Schrauwen},
  title = {{Factoring Variations in Natural Images with Deep Gaussian Mixture Models}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2014},
}
Figure