Rule Based Strategies for Image Interpretation

Terry E. Weymouth, J. S. Griffith, Allen R. Hanson, Edward M. RisemanAAAI 1983

Introduction

In the early 1980s, computer vision research encountered a significant bottleneck: the inherent ambiguity of low-level image data. A specific color or texture could represent vastly different objects depending on lighting, perspective, and environmental context. This 1983 classic addresses this by proposing a system that integrates top-down "world knowledge" with bottom-up image measurements to interpret complex suburban scenes.

Methodology

The proposed architecture relies on four primary components to resolve visual ambiguity and achieve reliable scene labeling.

Knowledge Network (Schemata)

The system utilizes a hierarchy of "schemata"—structured models of objects and scenes. A "House Scene" schema, for instance, expects a "House," which is further decomposed into "Roof" and "Wall" sub-parts. These models provide essential spatial constraints, such as the expectation that a roof typically appears above a wall, providing a framework for top-down interpretation.

Hierarchy of schemata representing world knowledge for a suburban house scene.

Rule-Based Voting and the Veto

Rather than employing binary classification, the system uses rules that assign a "vote" or score based on image features like color or intensity. These rules map measurements to a range of confidence values. A critical feature of this logic is the "veto" ($-\infty$), which allows the system to immediately reject a hypothesis if a measurement is fundamentally inconsistent with an object class, effectively filtering out spurious hypotheses.

The rule response function mapping image features to confidence scores, including veto thresholds.

Exemplar Strategy: Islands of Reliability

The authors observed that while object appearance varies significantly between different images (inter-image variation), it remains relatively consistent within a single image (intra-image variation). The system identifies high-confidence "exemplars"—the most prototypical instances of an object in a specific photo—and uses their features as "islands of reliability" to guide the identification of more ambiguous regions in the same scene.

Semantic Feedback and Resegmentation

Interpretation is not a linear process. When the system detects semantic conflicts (e.g., a region overlapping both "sky" and "wall" hypotheses), it triggers a targeted resegmentation. By adjusting parameters locally based on high-level knowledge, the system can resolve boundaries that were missed during initial processing.

Results and Discussion

The system was evaluated on a dataset of eight color suburban house scenes. Despite the small sample size, the results demonstrated the efficacy of combining semantic knowledge with adaptive measurements. The system successfully extracted natural elements (sky, grass, foliage) in five out of eight images and identified man-made structures in three. Notably, the semantic feedback loop successfully corrected segmentation errors, such as distinguishing house walls from the sky in cases where contrast was low.

Conclusion

This work highlights the necessity of "world knowledge" in computer vision. By treating the image as its own best reference through the exemplar strategy and employing a robust veto-based voting system, the authors provided a framework for handling the variability and ambiguity of real-world scenes. While limited by the manual tuning of rules and the computational constraints of the era, the principles of knowledge-directed segmentation remain foundational to the field.

Summary updated

BibTeX
@inproceedings{weymouth1983rule,
  author = {Terry E. Weymouth and J. S. Griffith and Allen R. Hanson and Edward M. Riseman},
  title = {{Rule Based Strategies for Image Interpretation}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {1983},
}

Loading related content...

Figure