Hierarchical Image Probability (H1P) Models

Clay Spence, Lucas C. ParraNeurIPS 1999

Introduction

Most computer vision models are "discriminative"—they calculate the probability of a class given an image ($Pr(class | image)$). While highly effective for classification tasks, these models function as "black boxes" regarding the underlying data itself. They cannot easily determine if an image looks "unnatural," nor are they suited for generative tasks like image compression or noise suppression.

To achieve these capabilities, researchers need a "generative" model that understands the probability distribution of the image itself ($Pr(image)$). However, modeling the distribution of all possible images is computationally daunting because pixels are highly dependent on one another across vast distances. Previous attempts, such as Markov Random Fields (MRFs), are often too slow, while others, like "flexible histograms," ignore the crucial fact that neighboring pixels share structural information from lower resolutions.

The Hierarchical Image Probability (HIP) model addresses this gap. It is a generative approach that factors the complex probability distribution of an entire image into a structured tree of conditional probabilities. By introducing hidden class labels at every level of an image pyramid, the model captures both fine-grained local textures and long-range structural dependencies.

Methods: The Architecture of HIP

The HIP model relies on three core methodological pillars to make generative image modeling tractable:

  1. Coarse-to-Fine Factoring: The authors demonstrate that any image distribution can be factored into a sequence of conditional distributions. They utilize a Gaussian pyramid—repeatedly blurring and downsampling an image—to represent the visual data at multiple resolutions.
  2. Hidden Variables: To capture long-range dependencies (e.g., the structural relationship between a plane's wing and its tail), the model attaches a hidden discrete label to every pixel at every level of the pyramid.
Quadtree structure of hidden variables across multiple resolution levels.

These labels form a quadtree structure (a belief network), where a "parent" label at a low resolution influences four "child" labels at the next higher resolution. This allows information to propagate efficiently across scales.

  1. The EM Algorithm: Training this massive tree of hidden variables is achieved using the Expectation-Maximization (EM) algorithm. The authors employ a "fine-to-coarse-to-fine" belief propagation procedure to efficiently update the model's parameters based on observed image data.

Key Insights

  • Exact Factoring is Possible: If the feature extraction process is invertible (like certain wavelet transforms), the probability of a whole image can be mathematically decomposed into a product of simpler, local conditional probabilities across scales.
  • Beyond Texture: While many hierarchical models excel at capturing repetitive textures, the HIP model goes further. By using hidden variables that propagate information down the tree, it successfully represents structured objects rather than just "visual noise."
  • The "Bow-Tie" Distribution: The model naturally reproduces the "bow-tie" shaped conditional distribution of features, a known phenomenon in image statistics. This confirms the HIP model successfully captures the non-stationary, multi-scale nature of real-world visual data.

Key Results and Performance

The authors evaluated the HIP model on two distinct types of imagery: Optical Aerial Imagery (for detecting aircraft at Logan Airport) and MSTAR (a public dataset of Synthetic Aperture Radar images for vehicle recognition).

Aircraft Detection: In the Logan Airport cross-validation study, the HIP model demonstrated exceptional performance. It achieved a mean Area Under the ROC Curve ($A_z$) of 0.94, significantly outperforming a multi-resolution neural network (HPNN), which scored 0.65 on the same dataset.

Comparison of ROC areas ($A_z$) between the HIP model and a multi-resolution neural network (HPNN) for aircraft detection.

SAR Target Recognition: On the MSTAR dataset, the HIP model's performance was comparable to the "flexible histogram" approach, which represented the state-of-the-art at the time. Notably, the HIP model was able to learn a single model for all aspect angles of a vehicle (BMP-2, BTR-70, and T-72), whereas previous methods often required a separate model for every 10-36 degrees of rotation.

Limitations and Future Work

Despite its successes, the HIP model has a few notable limitations:

  • Invertibility Constraint: The mathematical proof for factoring the image distribution relies heavily on the feature mapping being invertible. Relaxing this "strong" restriction for a wider variety of features remains an area for future work.
  • Computational Complexity: While more efficient than standard MRFs, running the EM algorithm on a massive quadtree is still computationally intensive.
  • Model Specificity: The current iteration assumes feature distributions are Normal (Gaussian) given a hidden label. While this makes the mathematics tractable, capturing the full complexity of real-world image features might eventually require more complex, non-Gaussian distributions.

Summary updated

BibTeX
@inproceedings{spence1999hierarchical,
  author = {Clay Spence and Lucas C. Parra},
  title = {{Hierarchical Image Probability (H1P) Models}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {1999},
}
Figure