Local figure–ground cues are valid for natural images

Charless C. Fowlkes, David R. Martin, Jitendra MalikJournal of Vision 2007 · DOI

TL;DR

This paper investigates how much information about figure-ground assignment can be extracted from local cues in natural images. By analyzing human-labeled natural scenes, they found that local cues like size, convexity, and relative position are statistically reliable predictors of figure-ground relationships, and developed a computational model that leverages these cues to achieve human-level performance on local judgments.

What Problem Does This Paper Address?

Figure-ground organization, the perceptual process of distinguishing an object (figure) from its background (ground), is fundamental to visual perception. While Gestalt psychologists emphasized global organizational principles, recent neuroscience findings suggest that local, bottom-up cues play a significant role, emerging very early in visual processing. This raises the question: how much information about figure-ground assignment is actually available from these local cues in the complex, natural world? The paper aims to quantify the ecological validity of these local cues and understand their relative importance in natural images.

Methods

The authors developed a computational framework to analyze local figure-ground cues. They defined three specific cues:

  • Size(p): The log ratio of the areas of the two neighboring regions around a contour point p.
  • LowerRegion(p): The cosine of the angle between the line connecting the centers of mass of the two regions and the vertical direction.
  • Convexity(p): The log ratio of the convexities of the two neighboring regions, where convexity is defined as the probability that a line segment connecting two points within a region lies entirely within that region.

These cues were computed within a local analysis window whose size was scaled relative to the contour length, making them approximately invariant to object distance. They then trained a simple logistic regression model to predict figure-ground assignments based on linear combinations of these cues.

Datasets

The primary dataset used was a collection of 200 natural images from the Berkeley Segmentation Dataset, which had been manually segmented. For these images, human observers were tasked with labeling figure-ground relationships for contours, creating a ground-truth dataset of figure-ground assignments. A subset of these labeled contours was used for both training and evaluating the computational model. Additionally, a separate experiment involved human subjects labeling local image patches (either with grayscale image content or just segment outlines) to assess the role of local luminance and shape cues.

Key Insights and Ideas

  • Ecological Validity of Gestalt Cues: The study provides strong empirical evidence that classic Gestalt cues like size, convexity, and the "lower region" principle are not arbitrary but are statistically reliable predictors of figure-ground relationships in natural scenes.
  • Local Cues are Powerful: Even without global context, local cues around a contour contain significant information for figure-ground assignment. The paper quantifies this information, showing that these local cues can predict figure-ground relationships with considerable accuracy.
  • Size is a Dominant Local Cue: Among the local cues studied, Size emerged as the most powerful predictor of figure-ground assignment.
  • Complementary Information: Combining cues like Size and LowerRegion yielded better performance than individual cues, suggesting they capture distinct aspects of the figure-ground relationship.
  • Luminance Content Matters: While shape cues are important, local luminance information (texture, shading, terminators) significantly boosts human performance in figure-ground judgments, often more than global shape reasoning.

Key Results

  • A computational model combining Size, LowerRegion, and Convexity cues achieved a 74% correct classification rate on local figure-ground judgments, approaching human performance on similar local tasks.
  • Human observers agreed on figure-ground/albedo labels for 83.9% of contour points in the ground-truth dataset.
  • When presented with only local shape information, human subjects achieved 69% correct classification, very close to the model's 68%.
  • The inclusion of local luminance content significantly improved human performance, leading to a >2x reduction in errors compared to shape-only presentations.

Limitations and Future Work

  • Model vs. Human Performance Gap: The model's 74% accuracy falls short of the 96% agreement level observed among human labelers for the ground truth. This gap is attributed to several factors not captured by the local model:
  • Global Reasoning: Humans likely integrate information across larger image regions, utilizing cues like symmetry and familiarity.
  • Junction Information: The model excludes image junctions, which provide crucial depth ordering information but are difficult to detect reliably.
  • Luminance Content: The initial model did not incorporate local luminance information, which was shown to be highly beneficial for human perception.
  • Future Directions: The authors suggest exploring how pooling local measurements might improve classification by propagating information from high-certainty zones. They also highlight the need to incorporate nonlocal and high-level cues, as well as luminance information, to bridge the gap between local and global performance. Further investigation into how luminance content aids figure-ground assignment, particularly through cues like terminators, shading, and texture, is also suggested.

Summary updated

Abstract

Figure-ground organization refers to the visual perception that a contour separating two regions belongs to one of the regions. Recent studies have found neural correlates of figure-ground assignment in V2 as early as 10-25 ms after response onset, providing strong support for the role of local bottom-up processing. How much information about figure-ground assignment is available from locally computed cues? Using a large collection of natural images, in which neighboring regions were assigned a figure-ground relation by human observers, we quantified the extent to which figural regions locally tend to be smaller, more convex, and lie below ground regions. Our results suggest that these Gestalt cues are ecologically valid, and we quantify their relative power. We have also developed a simple bottom-up computational model of figure-ground assignment that takes image contours as input. Using parameters fit to natural image statistics, the model is capable of matching human-level performance when scene context limited.
BibTeX
@article{fowlkes2007local,
  author = {Charless C. Fowlkes and David R. Martin and Jitendra Malik},
  title = {{Local figure–ground cues are valid for natural images}},
  journal = {Journal of Vision},
  year = {2007},
  doi = {10.1167/7.8.2},
}
Figure