Abstract
Modeling the distribution of natural images is challenging, partly because of strong statistical dependencies which can extend over hundreds of pixels. Recurrent neural networks have been successful in capturing long-range dependencies in a number of problems but only recently have found their way into generative image models. We here introduce a recurrent image model based on multi-dimensional long short-term memory units which are particularly suited for image modeling due to their spatial structure. Our model scales to images of arbitrary size and its likelihood is computationally tractable. We find that it outperforms the state of the art in quantitative comparisons on several image datasets and produces promising results when used for texture synthesis and inpainting.
Introduction
Modeling the distribution of natural images is a fundamental challenge in computer vision. The difficulty arises from the complex, long-range statistical dependencies inherent in visual data—for instance, the continuity of a long straight line or the repeating patterns of a texture. Traditional generative models often face a trade-off between computational tractability and the ability to capture these global contexts. Simple models like Gaussian Mixture Models (GMMs) are tractable but lack flexibility, while many deep models do not allow for the exact calculation of an image's "likelihood."
Furthermore, many autoregressive models predict pixels sequentially based on a small, local "causal neighborhood." This approach often fails to account for broader spatial context, limiting the model's ability to generate coherent large-scale structures. To address these issues, we introduce the Recurrent Image Density Estimator (RIDE), which leverages multi-dimensional Spatial LSTMs to capture long-range dependencies while remaining fully tractable.
Methods: The RIDE Architecture
The core innovation of RIDE is the Spatial LSTM (SLSTM). Unlike standard LSTMs designed for 1D sequences (like text), the SLSTM is specifically architected for 2D grids.
Spatial Information Propagation
For a pixel at coordinates $(i, j)$, the SLSTM integrates hidden states from its immediate neighbors to the left $(i, j-1)$ and above $(i-1, j)$. This directional flow allows information to propagate across the entire image. Consequently, the prediction for any given pixel can be influenced by the entire upper-left region, effectively providing a global context that traditional fixed-neighborhood models lack.
Output Layer: MCGSM
The hidden state of the SLSTM is fed into a Factorized Mixture of Conditional Gaussian Scale Mixtures (MCGSM). This component predicts a complex probability distribution for the target pixel. Natural image intensities are often multi-modal and heavy-tailed; the MCGSM allows RIDE to model these nuances more accurately than a single Gaussian output.
By stacking multiple SLSTM layers, RIDE factorizes the joint probability of an image into a product of conditional pixel probabilities. This ensures the model is fully tractable, allowing for the exact calculation of image log-likelihoods.
Key Insights and Quantitative Performance
A critical finding of this work is that simply increasing the neighborhood size in linear models yields diminishing returns. As shown in the comparison between RIDE and standard MCGSM, the SLSTM's advantage comes from its ability to perform non-linear transformations on spatial data over an effectively infinite context.
Experimental Results
RIDE was evaluated on several benchmarks, including BSDS300, van Hateren’s natural images, and the synthetic Dead Leaves dataset.
- State-of-the-Art Likelihoods: On BSDS300, RIDE outperformed previous leading models such as RNADE. For larger images, an ensemble of RIDE models achieved a log-likelihood of 3.899 bits per pixel, the best result reported at the time.
- Dead Leaves Dataset: RIDE successfully handled the sharp edges and complex occlusions of the Dead Leaves dataset, matching the performance of deep diffusion models.
- Qualitative Synthesis: The model demonstrated impressive capabilities in texture synthesis and inpainting, particularly for textures with strong directional components like wood grain or fabric.
Conclusion and Future Work
RIDE represents a significant step forward in generative image modeling by combining the memory-retention of LSTMs with a spatially-aware architecture. While the model excels at capturing long-range correlations, some limitations remain. The sequential nature of pixel-by-pixel prediction limits parallelization during generation, making it computationally expensive for high-resolution images. Additionally, while RIDE captures broad dependencies well, it can struggle with perfectly periodic patterns. Future research may explore architectures better suited for strict mathematical periodicity or optimizations to improve scaling to multi-megapixel resolutions.
Summary updated