Object Boundary Detection in Images using a Semantic Ontology

Anthony Hoogs, Roderic CollinsAAAI 2006

Abstract

Image segmentation is a fundamental challenge in computer vision, often hindered by the ambiguity between true object boundaries and visual clutter. This paper proposes a Semantic Distance Segmentation (SDS) framework that leverages WordNet, a hierarchical linguistic ontology, to provide high-level context for boundary detection. By calculating the semantic distance between image regions based on their visual-to-linguistic mappings, the system can identify boundaries even when low-level cues like color and texture are misleading. Evaluated on the Berkeley Segmentation Benchmark, the method demonstrates that semantic information offers a distinct and valuable signal for partitioning complex images.

Introduction: The Challenge of Semantic Boundaries

Traditional image segmentation algorithms typically rely on low-level features such as intensity, color, and texture gradients. However, these features often lead to two primary failure modes: over-segmentation, where visual patterns (like a striped shirt) are mistaken for multiple objects, and under-segmentation, where objects with similar visual properties (like a dark train against a dark bridge) are merged into a single region.

A challenging image from the BSDS dataset featuring a dark train on a stone bridge, illustrating the difficulty of boundary detection in low-contrast regions.

The authors argue that a "true" boundary is defined by a transition between two different semantic classes. The core difficulty lies in the fact that full object recognition is often more complex than the segmentation task itself. This research explores a middle ground: using sparse, high-level semantic hypotheses to guide pixel grouping without requiring perfect classification.

Methodology: Semantic Distance Segmentation (SDS)

The SDS approach follows a multi-stage pipeline designed to bridge the gap between pixels and concepts:

  1. Initial Partitioning: The process begins with an initial "oversegmentation" using the Mean-Shift algorithm to generate a set of candidate regions.
  2. Feature Extraction: Each region is characterized by a feature vector combining textons (quantized texture patterns) and RGB color histograms.
  3. WordNet Integration: These visual features are mapped to nodes in WordNet, a large hierarchical database of English nouns. Each node in the resulting "segmentation tree" represents a specific concept.
  4. Semantic Distance Calculation: To evaluate a potential boundary, the system maps the adjacent regions' features to their $K$-nearest semantic nodes in WordNet. A distance metric is then calculated based on the path length between these concepts in the hierarchy, weighted by their occurrence probability in training data.
  5. Probability Mapping: This semantic distance is converted into a boundary probability using a non-linear $tan^{-1}$ function.

Experimental Setup and Datasets

The researchers utilized the UC Berkeley Segmentation Benchmark (BSDS), which contains 300 images manually segmented by humans. To enable semantic evaluation, the authors manually augmented the dataset by labeling all 300 images with specific WordNet concepts for each segment, creating a unique, semantically-rich resource for the computer vision community.

Key Insights: Semantics vs. Visuals

A primary finding of the study is that semantic similarity is often orthogonal to visual similarity. For instance, a giraffe's coat and dry grass may appear visually identical but are semantically distant. This "hidden" cue allows the SDS method to detect boundaries that low-level algorithms might miss.

Furthermore, the authors observed that even "partial or erroneous" semantic hypotheses can provide sufficient evidence to distinguish between clutter and a true boundary. However, they also identified the "Pig Problem"—a limitation where two distinct objects of the same class (e.g., overlapping animals) have zero semantic distance, causing the algorithm to merge them.

Results and Discussion

The SDS method achieved an F-score of 0.59 on the test set, a notable improvement over the 0.54 baseline from the initial Mean-Shift segmentation. While a purely visual distance baseline scored 0.62, the lack of correlation between semantic and visual metrics suggests that an integrated model could yield superior performance.

Comparison of boundary detection results. The semantic approach (top) aims to filter out low-level visual clutter present in the initial oversegmentation (bottom) to identify meaningful object boundaries.

The study also established an upper bound F-score of 0.63 using ground-truth semantic labels, indicating that the performance is currently constrained by the quality of the initial oversegmentation.

Limitations and Future Directions

The model's performance was limited by data sparsity, with only 200 training images providing an average of 31 examples per WordNet node. Additionally, the "is-a" hierarchy of WordNet does not always align with visual "part-of" relationships (e.g., a steeple and a church). Future work could involve incorporating meronymy (part-whole relationships) and improving the initial segmentation pass to ensure critical boundaries are not lost before semantic analysis begins.

Summary updated

BibTeX
@inproceedings{hoogs2006object,
  author = {Anthony Hoogs and Roderic Collins},
  title = {{Object Boundary Detection in Images using a Semantic Ontology}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2006},
}
Figure