Cognitive Models for Computer Vision

G. Adorni, A. Boccalatte, M. Di ManzoCOLING 1982

TL;DR

This paper proposes a cognitive model for computer vision that integrates linguistic information with geometric representations to enable spatial inferencing. By formalizing spatial relationships and object descriptions, the authors aim to bridge the gap between natural language understanding and visual scene analysis, allowing robots to interpret and reason about their environment more effectively.

What Problem Does This Paper Address?

Interpreting visual scenes goes beyond simply identifying objects based on their geometric features. Humans leverage a rich understanding of object properties, their typical arrangements, and the context of their environment, often communicated through natural language. This paper addresses the challenge of imbuing computer vision systems with a similar cognitive capability. Specifically, it tackles how to translate linguistic descriptions of objects and their spatial relationships into a computational framework that supports sophisticated spatial reasoning, moving beyond purely geometric approaches. The authors highlight that understanding a scene involves not just recognizing shapes but also comprehending the functional and contextual aspects of objects, which are often conveyed through language.

Methods

The core of the proposed method lies in developing a cognitive model that integrates linguistic and visual information. This involves:

  • Object Representation: Objects are described at multiple levels of detail, from rough dimensions to detailed structures. Movable parts are explicitly modeled, and characteristic features (like a tabletop) are identified to aid in hypothesis generation.
  • Frame of Reference (FOR): Each object is associated with a redundant Frame of Reference (FOR) consisting of a major axis, an origin, and orthogonal axes. This FOR allows for a homogeneous representation of object shape and its spatial relations with the external world, simplifying reasoning in scenarios like "the ball is inside the box."
  • Spatial Relation Formalization: Linguistic prepositions and spatial relations (e.g., "above," "on," "behind," "in the middle of") are translated into quantitative geometric constraints using the FOR system. This allows for inferencing based on these relationships.
  • Generalized Cones: Solid objects are described using Generalized Cones, which can be interconnected with constraints on their relative positions and movements.

Datasets

Not discussed in the paper.

Key Insights and Ideas

  • Bridging Language and Vision: The paper argues that a clear separation between scene analysis and scene generation is artificial. A truly intelligent vision system needs to integrate knowledge from both domains, much like humans do.
  • Linguistic Cues as Geometric Constraints: Natural language descriptions, even simple ones like "on the table," carry significant quantitative geometric information that can be extracted and used to guide visual search and interpretation.
  • The Importance of Context and Prior Knowledge: Understanding spatial relationships often depends on the inherent properties of objects (e.g., a wall's shape) and typical object arrangements (e.g., pencils are found on tables). This suggests that a cognitive model needs to incorporate such knowledge.
  • Redundant FOR for Robustness: Associating a redundant Frame of Reference with each object, rather than relying solely on a minimal coordinate system, is proposed as a more psychologically plausible and computationally robust approach for representing spatial information.

Key Results

The paper presents a formalization of various spatial relations, demonstrating how they can be translated into computational terms. For instance, it provides conceptualizations for:

  • "Above": Defined by the relative Z-coordinates of points on two objects, with potential constraints on their horizontal projections.
  • "On": Similar to "above" but with an added assertion of physical contact and support against gravity.
  • "Behind": Conceptualized with respect to the FOR of the reference object or an external FOR, considering scenarios where one object might partially hide another.
  • "In the middle of": Defined by matching Z-coordinates and specific radial coordinates within a cross-section of the reference object.

The authors also outline how to handle relations along a path using curvilinear coordinates, enabling reasoning about sequential events.

Limitations and Future Work

The authors acknowledge several limitations and suggest future directions:

  • Quantifying Inequalities: The paper touches upon the need to quantify inequalities arising from spatial inferences but states this is beyond the scope of the current work due to space constraints.
  • Complex Spatial Relations: While basic prepositions are addressed, more complex or ambiguous spatial relations might require more sophisticated inferencing capabilities.
  • Variable Shapes and Actor Intentions: Representing objects with variable shapes and inferring an actor's future movements based on cognitive models are identified as open problems.
  • Linking Cognition and Image Processing: The paper emphasizes the need for better integration of cognitive processes with image-processing procedures.
  • Future Work: The authors state that this work is part of a larger project aiming to develop a cognitive background for an integrated vision-manipulation system, based on conceptual dependency. They also point to a reference for a classification of inferences.

Summary updated

BibTeX
@inproceedings{adorni1982cognitive,
  author = {G. Adorni and A. Boccalatte and M. Di Manzo},
  title = {{Cognitive Models for Computer Vision}},
  booktitle = {International Conference on Computational Linguistics},
  year = {1982},
}

Loading related content...

Figure