Generating more realistic images using gated MRF's

Marc'Aurelio Ranzato, Volodymyr Mnih, Geoffrey E. HintonNeurIPS 2010 · OpenReview

This paper, authored by Marc'Aurelio Ranzato, Volodymyr Mnih, and Geoffrey Hinton, addresses a long-standing embarrassment in generative modeling: the fact that even our best probabilistic models of natural images often produce samples that look like random noise when viewed at high resolutions.

By augmenting gated Markov Random Fields (MRFs) with "mean hidden units" that explicitly model pixel intensities alongside covariances, and combining this with a "tiled-convolutional" weight-sharing scheme, the authors produce high-resolution samples that exhibit the smooth regions and sharp boundaries characteristic of real-world scenes.

The Problem with Standard Generative Models

While probabilistic models like MRFs are highly effective for tasks like denoising, they historically fail as true generative models. If you ask a standard model to "draw" a high-resolution image, the result usually lacks the long-range structure of natural scenes, appearing instead as a mess of high-frequency patterns.

The authors identify two primary culprits for this failure:

  1. Ignoring the Mean: Most existing models (like the Product of Student's t, or PoT) focus almost exclusively on modeling the relationships between pixels (covariances) while assuming the data is zero-mean. In high-resolution images, overlapping patches have different mean intensities. Failing to model this leads to "random-looking" filters.
  2. Inefficient Weight Sharing: Standard convolutional weight sharing (where a filter is applied at every single pixel) creates massive redundancy. This forces the model to learn independent-looking, high-frequency filters to compensate for the high correlation between overlapping latent variables.

Methods: Rethinking Gated MRFs

To solve these issues, the researchers introduce two key innovations to the gated MRF framework:

1. Mean Hidden Units (mPoT and mcRBM)

The authors add a second set of latent variables to the model. While the first set "gates" or modulates the pairwise interactions between pixels (capturing texture and edges), the second set explicitly sets the mean intensities of the pixels. This allows the model to represent both the "what" (structure) and the "how bright" (intensity) of an image patch. By subtracting the relevant mean via latent variables, the covariance units are freed up to capture the actual structural data.

2. Tiled-Convolutional (TConv) Weight Sharing

Instead of the standard convolutional approach (stride of 1), they propose a scheme where filters tile the image without overlapping with copies of themselves. However, different filters do overlap with each other.

Diagrams illustrating different weight-sharing schemes, including the proposed Tiled-Convolutional approach.

This reduces spatial redundancy and allows the model to learn localized, oriented Gabor-like features—the building blocks of natural vision. It acts as a middle ground between "Global" models (which don't scale) and "Conv" models (which are too redundant).

Key Results and Insights

The authors evaluated their approach using the Berkeley Segmentation Dataset (for high-resolution generation), MNIST, and CIFAR-10. They argue that drawing samples is the "acid test" for whether a model truly understands the statistics of natural images, as a model that is good at denoising isn't necessarily a good generative model.

Qualitative and Quantitative Improvements

Samples from the new mPoT-TConv model show clear, smooth regions and sharp boundaries. In contrast, samples from standard PoT models look like unstructured grey noise.

Samples generated by the mPoT-TConv model, exhibiting smooth regions and sharp boundaries.

Using KL divergence to compare the statistics of generated images versus natural images, the authors found that models with mean units were significantly closer to "real" image statistics than those without.

Efficiency in Recognition

In MNIST and CIFAR-10 tests, the "Local" and "TConv" schemes reached higher accuracy much faster than standard "Global" or "Conv" models. For instance, the "Local" scheme was roughly 4 times faster than the "Global" scheme while achieving better performance.

Limitations and Future Work

Despite the significant improvements in generating realistic textures and boundaries, the authors acknowledge that these models are still "shallow." While they capture local structure well, they do not yet capture the global composition of a scene (e.g., a "face" or a "house").

Suggested next steps involve integrating these gated MRFs into deeper, hierarchical architectures to capture longer-range dependencies, and using the model's improved generative power to better handle missing values or highly ambiguous sensory data.

Summary updated

Abstract

Probabilistic models of natural images are usually evaluated by measuring performance on rather indirect tasks, such as denoising and inpainting. A more direct way to evaluate a generative model is to draw samples from it and to check whether statistical properties of the samples match the statistics of natural images. This method is seldom used with high-resolution images, because current models produce samples that are very different from natural images, as assessed by even simple visual inspection. We investigate the reasons for this failure and we show that by augmenting existing models so that there are two sets of latent variables, one set modelling pixel intensities and the other set modelling image-specific pixel covariances, we are able to generate high-resolution images that look much more realistic than before. The overall model can be interpreted as a gated MRF where both pair-wise dependencies and mean intensities of pixels are modulated by the states of latent variables. Finally, we confirm that if we disallow weight-sharing between receptive fields that overlap each other, the gated MRF learns more efficient internal representations, as demonstrated in several recognition tasks.
BibTeX
@inproceedings{ranzato2010generating,
  author = {Marc'Aurelio Ranzato and Volodymyr Mnih and Geoffrey E. Hinton},
  title = {{Generating more realistic images using gated MRF's}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2010},
  url = {https://openreview.net/forum?id=HJZ0-Ybubr},
}
Figure