Natural images are complex tapestries composed of distinct local textures such as grass, brick, or fabric. Modeling these textures effectively is a fundamental challenge in computer vision. In this work, Kivinen and Williams from the University of Edinburgh investigate the generative capabilities of Boltzmann Machines, specifically evaluating their ability to synthesize the intricate patterns found in the natural world.
Problem Statement
While deep learning models excel at image classification, generating realistic, high-resolution textures remains difficult. Previous state-of-the-art models, like the mean Product of Student-t (mPoT), were often trained on general natural image patches and frequently failed to produce convincing textured regions. This research addresses two primary questions:
- How well do these models perform when trained specifically on textures?
- Can a single, efficient model generate multiple different textures by sharing visual filters?
Methodology
The authors explore three variations of Boltzmann Machines, utilizing tiled-convolutional weight sharing to scale to large images:
- Tm (Gaussian RBM): Focuses on the conditional means of pixel distributions.
- TPoT (Product of Student-t): Focuses on pixel covariances.
- TmPoT: A hybrid model combining both mean and covariance components.
The core innovation is the Multi-Tm. Instead of training isolated models for each texture, the Multi-Tm utilizes a shared dictionary of filters. To switch between textures, the model employs texture-specific hidden unit biases. These biases act as high-level switches, activating relevant visual features for a specific pattern while suppressing others.
Key Insights: The Power of the Mean and Feature Sharing
A significant finding was that the Gaussian RBM (Tm), which models only the conditional mean, performed nearly as well as the more complex hybrid mPoT. Conversely, the covariance-only model (TPoT) struggled with texture synthesis. This suggests that capturing local average structure is more critical for textures than modeling complex third-order interactions.
Furthermore, the Multi-Tm demonstrated that textures share common visual primitives. By sharing filters, the model learns a more robust feature set. The authors also showed that by spatially varying these biases, the model can transition smoothly between different textures within a single image.
Experimental Results
The models were evaluated on the Brodatz texture database using the Texture Similarity Score (TSS), a metric based on normalized cross-correlation. Both Tm and TmPoT achieved high scores (often >0.90), outperforming previous methods.
The Multi-Tm's performance scaled positively with the number of shared features, matching or exceeding the quality of individual models. In inpainting tasks, the Multi-Tm successfully maintained global structure and local detail, comparable to non-parametric "copy-paste" methods.
Conclusion and Future Directions
While the tiling scheme occasionally introduced boundary artifacts, the "bias-switching" mechanism offers a promising path toward comprehensive natural image models. Future work could involve deeper architectures where high-level layers automatically segment scenes and adjust lower-level biases to generate appropriate textures for different objects, such as sky, grass, or stone.
Summary updated