Exact feature probabilities in images with occlusion

Xaq Pitkow

TL;DR

This paper provides the first exact mathematical solution to the "dead leaves model," a naturalistic image model where scenes are built from an infinite stack of occluding objects. By deriving the joint probability distributions of image features without approximation, the author explains why natural images exhibit specific non-Gaussian statistics, such as "heavy-tailed" distributions and "bowtie-shaped" dependencies between edges.

What Problem Does This Paper Address?

To understand how our brains (or AI systems) process visual information, we need to understand the statistical structure of the world we see. Most traditional models treat images as a linear superposition of features—essentially adding waves or patterns on top of each other (like PCA or ICA).

However, the real world doesn't work by addition; it works by occlusion. When one object moves in front of another, it doesn't blend with it; it hides it. This creates a strong nonlinearity that linear models fail to capture. While the "dead leaves model" (an infinitely deep stack of random shapes) has been used to simulate occlusion for decades, researchers previously had to rely on empirical sampling to understand its statistics. This paper solves the model analytically, providing a principled way to relate observable pixels to unobservable object properties like depth and boundaries.

Methods

The author develops a recursive mathematical framework to calculate the object membership function (denoted by the Hebrew letter $\aleph$, or mem).

  1. Membership Probabilities: For any $N$ pixels, the model calculates the probability that they belong to the same object or different objects. Because objects are independent, the joint probability of intensities factorizes once you know which pixels share an object.
  2. Recursive Solution: The paper derives a recursion relation (Equation 3) that determines the probability of a specific configuration by considering the topmost object and then recursively solving for the "deeper" layers.
  3. Geometric Integration: To handle arbitrary shapes (like ellipses), the author uses the divergence theorem to convert complex 2D area integrals into simpler 1D contour integrals along object boundaries.
  4. Feature Transformation: These pixel-level probabilities are then transformed into feature-level probabilities (e.g., wavelet coefficients) to compare the model against real-world image data.

Datasets

  • Simulated Data: Images generated using the "dead leaves model" with various shape ensembles (circles and ellipses) and size distributions (power-law/scale-invariant).
  • Natural Images: The author uses van Hateren’s image database, a standard benchmark in vision science containing high-quality calibrated images of natural scenes, to validate that the model's analytical predictions match real-world statistics.

Key Insights and Ideas

  • The Origin of Heavy Tails: The "heavy-tailed" (non-Gaussian) distribution of spatial derivatives in natural images is often attributed to the presence of edges. This paper shows it actually arises from a combination of scale invariance (objects existing at all sizes) and sub-pixel integration (the fact that a single pixel averages light over a finite area).
  • Geometry Dictates Distribution Shapes: The "diamond," "pillow," and "bowtie" shapes seen in joint distributions of wavelet coefficients are not random. They are direct consequences of how object boundaries rotate and overlap with localized filters. For example, a boundary aligned with one wavelet filter necessarily aligns poorly with an orthogonal one, creating a "diamond" envelope.
  • Depth is Essential for Contours: To explain why certain edges appear to belong to the same contour (contour completion), the model must account for relative depth (border ownership). Without a depth constraint, the statistics don't match human perception or natural scene data.

Key Results

  • Exact Match to Empirical Data: The analytical solution perfectly reproduces the peculiar shapes of joint and conditional distributions found in natural images (e.g., the "bowtie" dependency where the variance of one feature increases with the amplitude of a neighbor).
  • Contour Prediction: By defining a "common cause" for edges that includes depth constraints, the model's likelihood ratios for edge co-occurrence match the empirical results of Geisler et al., which are known to predict human contour-grouping judgments.
  • Generalization: The framework is shown to be universal for any "exclusion" process where one cause dominates at each point (like craters on the moon), not just visual occlusion.

Limitations and Future Work

  • Computational Complexity: The number of possible membership configurations grows according to Bell’s number, which increases faster than exponentially. This limits exact calculations to small patches of about a dozen pixels.
  • Simplified Shapes: The current derivation primarily uses ellipses. While these capture many natural statistics, real-world objects have more complex, non-convex geometries.
  • Independence Assumption: The model assumes object attributes (size, color, position) are independent. In reality, natural scenes have global structures like gravity, lighting direction, and semantic groupings that the dead leaves model ignores.
  • Future Directions: The author suggests extending the model to include 3D cues like binocular disparity and perspective, which could lead to a more complete "neural code" for how the brain interprets natural scenes.

Summary updated

Abstract

To understand the computations of our visual system, it is important to understand also the natural environment it evolved to interpret. Unfortunately, existing models of the visual environment are either unrealistic or too complex for mathematical description. Here we describe a naturalistic image model and present a mathematical solution for the statistical relationships between the image features and model variables. The world described by this model is composed of independent, opaque, textured objects, which occlude each other. This simple structure allows us to calculate the joint probability distribution of image values sampled at multiple arbitrarily located points, without approximation. This result can be converted into probabilistic relationships between observable image features as well as between the unobservable properties that caused these features, including object boundaries and relative depth. We show that the image model is sufficient to explain a wide range of natural scene properties. Finally, we discuss the implications of this description of natural scenes for the study of vision.
BibTeX
@article{pitkow2010exact,
  author = {Xaq Pitkow},
  title = {{Exact feature probabilities in images with occlusion}},
  journal = {Journal of Vision},
  year = {2010},
  doi = {10.1167/10.14.42},
}

Loading related content...

Figure