Grouping Contours Via a Related Image

Praveen Srinivasan, Liming Wang, Jianbo ShiNeurIPS 2008

Introduction

In computer vision, contours—ordered sets of edge pixels—are excellent for representing shape but often lack "spatial support." Because they are thin structures, they do not inherently indicate which pixels belong to the same object. Conversely, region segmentation provides spatial support but often lacks the fine-grained structural detail of contours.

The challenge lies in grouping these "bottom-up" detected contours into larger, object-level parts. Doing this in a single image is notoriously difficult due to clutter and occlusion. To address this, researchers propose a robust framework that leverages a related image (e.g., a stereo pair, a video frame, or a visually similar object) to provide the necessary cues. If a set of contours moves together in a video or maintains a consistent disparity in stereo, they likely belong to the same object. By treating contour grouping as a "context selection" problem solved via Linear Programming and Graph Cuts, this approach demonstrates a unified method that works across stereo, motion, and similarity modalities.

Methodology

The proposed multi-stage pipeline moves from local features to global grouping through three primary steps:

  1. Candidate Transformations: The system first identifies potential alignments between the two images using SIFT features and RANSAC to propose transformations (such as rotation, scale, and translation).
  2. Context Selection: For a given transformation, the algorithm does not simply match points. Instead, it solves a selection problem: which subset of contours in Image A and Image B, when taken together, form the most consistent shape? The authors use Shape Contexts (log-polar histograms of edge distributions) but modify the standard process. Rather than using all edges, they employ an Integer Linear Program (LP) to select a subset of contours that minimizes a matching cost. This cost balances similarity (how well the shapes match) and maximality (including as many contours as possible).
  3. Global Grouping via Graph Cuts: Once the optimal contexts are found, the final grouping is formulated as an inference problem in a Markov Random Field (MRF). The $\alpha-\beta$ swap graph cut algorithm is used to assign contours to groups, ensuring "common fate" (contours moving or transforming together) and "good continuation" (overlapping contours staying together).

Datasets and Evaluation

The authors evaluated their method on several types of data to prove its versatility. For stereo and motion tasks, they used data captured from a car-mounted stereo rig (1m baseline) providing real-world street scenes. For similarity tasks, they utilized image pairs from the Caltech 101 dataset, which contains various object categories. In all cases, input contours were generated using an automated contour extraction method rather than relying on hand-labeled edges.

Key Insights and Results

A major insight of this work is that context fixes matching. Traditional local matching is sound but often fails in clutter. By selecting a specific context (a subset of contours), the algorithm effectively ignores "distractor" edges that would otherwise corrupt the shape descriptor. Furthermore, relaxing the hard "Integer" selection problem into a standard Linear Program makes this context selection computationally feasible.

Grouping contours across stereo, motion, and similarity modalities.

Because the method relies purely on shape matching under a transformation, it achieves a unified modality. It does not care if the relationship is stereo, motion, or visual similarity—a significant departure from specialized algorithms built for only one specific type of image pair.

Qualitatively, the method produces much cleaner, more "object-like" groupings than baselines lacking context selection. In street scenes, it successfully groups the internal contours of cars and building facades into single units. In similarity tests (e.g., matching two different faces), context selection allows the model to ignore background edges and focus purely on shared geometry. Additionally, in stereo pairs, the resulting groups show highly consistent disparity (depth), even though the algorithm infers this entirely through shape-consistent grouping rather than explicit depth information.

Limitations and Future Work

Despite its robustness, the approach has a few limitations:

  • Computational Complexity: While the LP relaxation is efficient, the unoptimized MATLAB implementation still takes several minutes per image pair, keeping it far from real-time application.
  • Transformation Constraints: The current model primarily handles similarity transforms. Extending this to non-linear or piecewise linear transformations is necessary to handle articulated objects, like a person walking.
  • Learning Parameters: The framework currently uses hand-tuned parameters for its cost functions. Future work could involve learning which shape configurations are most reliable for matching to further improve grouping results.

Summary updated

BibTeX
@inproceedings{srinivasan2008grouping,
  author = {Praveen Srinivasan and Liming Wang and Jianbo Shi},
  title = {{Grouping Contours Via a Related Image}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2008},
}
Figure