Cue Integration for Figure/Ground Labeling

Xiaofeng Ren, Charless C. Fowlkes, Jitendra MalikNeurIPS 2005 · OpenReview

This paper introduces a Conditional Random Field (CRF) model that solves the classic "figure/ground" problem by integrating low-level edges, mid-level continuity, and high-level object shape. By moving away from pixel-level grids to a scale-invariant "triangulated" representation, the authors demonstrate that mid-level cues like junctions are surprisingly powerful for accurate image segmentation.

The Figure/Ground Problem

In computer vision, "figure/ground organization" is the task of determining which parts of an image belong to a foreground object and which belong to the background. While humans do this effortlessly, it is a complex puzzle for machines.

Historically, researchers used Markov Random Fields (MRFs) to model this, but these models were usually tied to individual pixels. This created two major headaches:

  1. Computational Expense: Processing millions of pixels is slow.
  2. Scale Blindness: Pixel-level models struggle to recognize structures like junctions or long curves that exist across different scales.

The authors aim to bridge the gap between low-level image features (like color and edges) and high-level knowledge (what a specific object, like a horse, looks like) using a unified probabilistic framework.

Methodology

The researchers propose a three-stage approach that fundamentally changes how the image is represented and processed.

The four stages of the proposed method: (1) original image, (2) edge detection, (3) Constrained Delaunay Triangulation, and (4) the CRF model structure.
  1. A New Representation: Instead of a pixel grid, they use Constrained Delaunay Triangulation (CDT). As seen in the figure above, they first detect edges (2), simplify them into straight lines, and finally use triangulation to fill in the gaps (3). This creates a graph of "edges" and "triangular regions" that is much smaller and more manageable than a dense pixel grid.
  2. The CRF Model: They build a Conditional Random Field over three types of variables (illustrated in panel 4):
  3. Edges ($X_e$): Is this line a true boundary?
  4. Regions ($Y_s, Y_t$): Is this triangle part of the foreground or background?
  5. Pose ($Z$): Where is the object located in the scene?
  6. Multi-Level Cues: The model's "energy function" combines three tiers of information:
  7. Low-level: Local edge contrast and similarity in brightness/texture between adjacent regions.
  8. Mid-level: "Curvilinear continuity" (do edges line up smoothly?) and "Closure" (do the boundaries form a solid shape?).
  9. High-level: "Texture familiarity" (does this look like horse skin?) and "Global shape" (does this match a general horse silhouette?). They also use "shapemes"—local geometric descriptors—to identify specific parts like legs or backs.

The parameters of this model are learned from human-labeled data using Loopy Belief Propagation and gradient descent.

Datasets and Evaluation

The study primarily uses the Borenstein et al. Horse Dataset, consisting of 344 grayscale images. The dataset is split in half for training and testing. Human-marked segmentations serve as the "ground truth" for both teaching the model and evaluating its accuracy.

Key Findings and Results

The authors evaluated their model using Precision-Recall curves and the F-measure (a balance of the two). The probabilistic framework allowed the authors to "surgically remove" specific cues to see exactly how much each one contributed to the final result:

  • Baseline ($Pb$): Using only low-level boundary detection yielded an F-measure of 0.54.
  • Adding Mid-level ($Pb+M$): Improved the score to 0.56.
  • Adding High-level ($Pb+H$): Jumped to 0.62.
  • Full Model ($Pb+M+H$): Achieved the best performance at 0.66.

Key Insights: * The Power of Mid-Level Cues: One of the most significant findings is that mid-level cues (like junctions and continuity) provide a substantial boost in performance, even when high-level shape knowledge is already available. * Duality of Edges and Regions: By modeling both edges and regions simultaneously, the model ensures consistency—if an edge is "on," the regions on either side should have different labels (one figure, one ground). * Scale Invariance: Because the CDT representation is based on geometry rather than pixels, the model can handle objects of different sizes and orientations much more robustly than traditional methods.

Qualitatively, the model proved capable of suppressing "distractor" gradients in the background (like grass or fences) while boosting faint, low-contrast edges that actually defined the horse's shape.

Limitations and Future Work

While highly effective, the current experiments are limited to a single object class (horses). Although the authors believe the cues will scale, the high-level shape templates are currently category-specific. Additionally, the model uses a relatively simple grid for object location. Future work could involve more complex pose estimation to handle highly articulated movements and extending this model to handle multiple object categories simultaneously, moving toward a more general-purpose scene understanding tool.

Summary updated

Abstract

We present a model of edge and region grouping using a conditional random field built over a scale-invariant representation of images to integrate multiple cues. Our model includes potentials that capture low-level similarity, mid-level curvilinear continuity and high-level object shape. Maximum likelihood parameters for the model are learned from human labeled groundtruth on a large collection of horse images using belief propagation. Using held out test data, we quantify the information gained by incorporating generic mid-level cues and high-level shape.
BibTeX
@inproceedings{ren2005cue,
  author = {Xiaofeng Ren and Charless C. Fowlkes and Jitendra Malik},
  title = {{Cue Integration for Figure/Ground Labeling}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2005},
  url = {https://openreview.net/forum?id=r1NNN_Wu-r},
}
Figure