Multimodal Cognitive Architecture: Making Perception More Central to Intelligent Behavior

B. ChandrasekaranAAAI 2006

Abstract

Traditional AI architectures often treat perception as a peripheral input module that translates sensory data into symbolic strings for a central reasoning engine. This paper proposes a shift toward a multimodal cognitive architecture where internal perceptual and kinesthetic images are integral to the reasoning process. By incorporating internal perceptual states and multimodal memory, agents can perform qualitative abstractions and solve spatial problems that are computationally expensive or impossible for purely symbolic systems.

Introduction

For decades, dominant cognitive architectures like Soar and ACT-R have relied on the "Language of Thought" hypothesis, positing that thinking is essentially the manipulation of predicate-symbolic strings (e.g., ON(BlockA, BlockB)). However, this approach faces significant hurdles in qualitative abstraction. Converting a rich visual scene into symbols often discards vital metric and quantitative information. For instance, if a robot only knows Left(A, B) and Left(B, C), it cannot inherently determine if A is further from B than B is from C without pre-defined relations. Humans resolve such ambiguities by "seeing" the answer in a mental image. This research seeks to integrate this "internal perception" into the core of AI reasoning.

The Multimodal Architecture

The proposed framework broadens the cognitive state beyond symbolic strings to include several key components:

  • External Perceptual Systems (EPS): Modules processing raw sensory data.
  • Internal Perceptual States (IPS): Organized perceptual experiences (e.g., a visual image of a shape) that serve as the output of the EPS.
  • Multimodal Memory: A long-term storage system capable of retrieving perceptual components alongside symbolic facts.
  • Perceptual Operators: Routines that "examine" an IPS to extract symbolic information on demand.
The proposed multimodal cognitive architecture featuring Internal Perceptual States (IPS) and Multimodal Memory.

A core implementation of this theory is Diagrammatic Representation and Reasoning (DRS), where diagrams are treated as configurations of points, curves, and regions. This allows the system to execute "action routines" (e.g., drawing a non-intersecting line) and "perceptual routines" (e.g., checking point-in-region containment) to solve complex spatial problems.

Key Insights

The shift toward multimodal cognition offers several advantages over traditional symbolic-only models:

  • Perception as Cognition: Perception is not just a front-end; it is a form of thinking. Applying perceptual operators to internal images allows agents to "infer" relations never explicitly programmed.
  • Compositionality: Contrary to the belief that images lack structure, perceptual primitives (like 3D shapes) can be composed to create novel internal scenes.
  • Representational Economy: Instead of storing every possible symbolic relation, an agent can store a single perceptual image and query it for specific relations as needed.
  • Associative Evocation: Thinking can move fluidly between modalities, where a symbolic word like "apple" evokes visual and auditory IPS useful for problem-solving.

Demonstration and Results

The paper demonstrates a bimodal (symbolic + diagrammatic) system using a modified "Blocks World" scenario. Unlike traditional symbolic systems, this architecture does not require a complete symbolic description of every state.

A bimodal representation in the Blocks World task, combining symbolic predicates with diagrammatic states.

By using a diagrammatic component, the system can "see" if a block is on the table and use action routines to "move" blocks, updating both the internal image and the symbolic state simultaneously. This approach provides a computational path to reconcile the "picture view" of mental imagery with the "structural view" required for systematic thought.

Limitations and Future Work

While promising, several challenges remain. The specifics of how an agent learns to store and retrieve complex perceptual parameters are still being explored. Additionally, while the framework is theoretically multimodal, current implementations focus heavily on the diagrammatic (visual) modality; extending this to kinesthetic or auditory reasoning is a future goal. Finally, while the model aligns with neuroscientific theories, it remains a computational abstraction rather than a direct biological simulation.

Summary updated

BibTeX
@inproceedings{chandrasekaran2006multimodal,
  author = {B. Chandrasekaran},
  title = {{Multimodal Cognitive Architecture: Making Perception More Central to Intelligent Behavior}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2006},
}
Figure