Abstract
An image pattern can be represented by a probability distribution whose density is concentrated on different low-dimensional subspaces in the high-dimensional image space. Such probability densities have an astronomical number of local modes corresponding to typical pattern appearances. Related groups of modes can join to form macroscopic image basins that represent pattern concepts. Recent works use neural networks that capture high-order image statistics to learn Gibbs models capable of synthesizing realistic images of many patterns. However, characterizing a learned probability density to uncover the Hopfield memories of the model, encoded by the structure of the local modes, remains an open challenge. In this work, we present novel computational experiments that map and visualize the local mode structure of Gibbs densities. Efficient mapping requires identifying the global basins without enumerating the countless modes. Inspired by Grenander's jump-diffusion method, we propose a new MCMC tool called Attraction-Diffusion (AD) that can capture the macroscopic structure of highly non-convex densities by measuring metastability of local modes. AD involves altering the target density with a magnetization potential penalizing distance from a known mode and running an MCMC sample of the altered density to measure the stability of the initial chain state. Using a low-dimensional generator network to facilitate exploration, we map image spaces with up to 12,288 dimensions (64 $\times$ 64 pixels in RGB). Our work shows: (1) AD can efficiently map highly non-convex probability densities, (2) metastable regions of pattern probability densities contain coherent groups of images, and (3) the perceptibility of differences between training images influences the metastability of image basins.
Introduction
Modern generative models, such as those based on neural networks, represent image patterns as probability densities in high-dimensional spaces. These landscapes are incredibly complex, containing an "astronomical" number of local modes, or peaks. While each mode might represent a slightly different version of an image, related modes often cluster together to form larger "basins of attraction" that correspond to a single concept, such as a specific handwritten digit.
The challenge lies in the vastness of these landscapes. Standard exploration tools often get trapped in local irregularities, making it difficult to understand the global structure of what a model has learned. This paper introduces Attraction-Diffusion (AD), a novel MCMC-based "telescope" designed to map the macroscopic structure of these high-dimensional, highly non-convex image probability landscapes. By measuring the metastability of local modes, the authors can group millions of individual image variations into coherent "galaxies" that represent distinct visual concepts.
Methodology: Attraction-Diffusion (AD)
The core innovation is the Attraction-Diffusion (AD) technique. The method alters the target energy function $E(X)$ by adding a "magnetization" term: $$E_{T, \alpha, X^*}(X) = E(X)/T + \alpha\|X - X^*\|_2$$ In this formulation, $X^*$ is a known local minimum, and $\alpha$ represents the strength of a "magnetic field" pulling the sampler toward it. By tuning $\alpha$ and the temperature $T$, the researchers can measure the metastability of a region. If a sampler can escape a basin even with a weak pull toward the center, that basin is considered unstable. Conversely, if it remains trapped, the basin represents a strong "memory."
This technique is integrated into the ADELM (Attraction-Diffusion Energy Landscape Mapping) algorithm, which identifies local minima and uses AD trials to determine if new minima "diffuse" into previously discovered basins. To handle the prohibitively large 12,288-dimensional image space (64x64 RGB), the authors utilize a low-dimensional generator network to facilitate exploration and "clean" the landscape of unrealistic low-energy regions.
Experimental Results
The study evaluates ADELM across several diverse landscapes, ranging from physics models to natural images.
Validation and Concept Grouping
On the SK spin-glass model, a 100-state physics benchmark, ADELM successfully captured macroscopic features, identifying stable basins that matched ground-truth symmetric properties. When applied to the MNIST dataset, the algorithm correctly grouped different tilts and styles of handwritten digits into coherent basins. For instance, it demonstrated that different versions of the digit "1" merge at lower energy levels than the digit "0," which remains more distinct.
Information Scaling
A key insight of the paper is "Information Scaling," studied using texture patches of ivy leaves at multiple scales. The results show that as the scale of an image increases, the landscape structure changes fundamentally.
For "textons" (small, recognizable features), the landscape contains many stable, distinct basins. However, for "textures" (large-scale repetitive patterns), these basins merge into a single wide, shallow region. This reflects human visual intuition: as images become more textured and less distinct, individual variations become harder to distinguish, leading to a merging of their corresponding energy basins.
Discussion and Conclusion
The Attraction-Diffusion method provides a significant leap in efficiency, identifying major basins within the first 100 iterations, whereas previous methods required millions. By prioritizing metastability over traditional barrier height, ADELM offers a more natural way to define "closeness" in complex probability landscapes.
While the use of generator networks helps manage high dimensionality, mapping very high-resolution spaces without such aids remains a challenge. Future work may explore hierarchical relationships between image memories—such as how "eyes" and "ears" combine to form a "face"—and the definition of compositional "dictionaries" for image patches. Ultimately, this "telescope" reveals how human perceptibility influences the stability of learned image memories within neural networks.
Summary updated