A Robotic Model of Human Reference Resolution

Matthias Scheutz, Virgil Andronache, Kathleen M. EberhardAAAI 2004

Introduction

Traditional artificial intelligence often approaches natural language processing (NLP) through a "wait-and-see" strategy. In this paradigm, a system listens to an entire sentence before attempting to parse its meaning and resolve references to objects in the environment. However, psychological evidence indicates that humans process language incrementally, narrowing down potential visual targets in real-time as words are spoken. This paper presents a distributed robotic architecture that mimics this human-like incremental reference resolution, improving efficiency and responsiveness in dynamic physical environments.

System Architecture

The proposed model is implemented using the APOC (Affecting People and Objects in Context) framework, a distributed architecture designed for complex agents. Unlike monolithic systems, this approach utilizes specialized nodes that operate concurrently to handle speech recognition, visual tracking, and spatial reasoning.

The distributed robotic architecture based on the APOC framework.

The core of the system involves several key components:

  • Linguistic Processing: Real-time speech recognition is handled by Sphinx 2. A "Word Analysis" node ensures serial processing, while "Syntax Analysis" triggers "Referent Analysis" as soon as relevant words are identified.
  • Incremental Referent Analysis: This is the central innovation. As words arrive, the system maintains a set of possible referents ($PR$). For instance, hearing the word "red" immediately populates $PR$ with all red objects in the scene. If $PR$ becomes a singleton, the reference is resolved instantly, even before the sentence concludes.
  • Visual and Spatial Tracking: A multi-layered visual system performs color blob detection and tracking. A dedicated spatial analysis node handles relational terms (e.g., "on" or "next-to"), allowing the robot to track complex relationships between objects.

Experimental Evaluation

The system was tested using an ActivMedia Pioneer Peoplebot in a controlled "Blocks World" environment. The experiments focused on two primary scenarios designed to test the limits of incremental resolution.

Comparison of the 1-Referent and 2-Referent tasks in the Blocks World domain.
  1. 2-Referent Task: The scene contains two red blocks. The phrase "the red block" remains ambiguous until further descriptive information is provided, requiring the robot to maintain multiple hypotheses.
  2. 1-Referent Task: The scene contains only one red block. Here, the robot can uniquely identify the target early in the sentence, demonstrating "early commitment."

Discussion and Key Insights

The research highlights several critical advantages of incrementalism over traditional backtracking methods. By resolving references as soon as the visual context allows, the robot avoids the "combinatorial explosion" of parse trees that often hinders traditional syntactic-first approaches.

Furthermore, the model successfully replicated the "Garden Path" effect observed in humans. In the 1-referent task, because the robot identifies the target immediately, it may treat subsequent, redundant information (e.g., "on the blue block") as part of a new instruction or ignore it entirely. This mirrors human cognitive behavior where premature closure of a reference can lead to confusion if additional, unexpected details follow.

Conclusion

The authors successfully demonstrated a real-time, parallel implementation of a cognitive model for reference resolution. While the current "Blocks World" environment is simplified, the results show that incremental processing significantly enhances a robot's ability to interact naturally with humans. Future work will focus on expanding the linguistic breadth of the system and testing its performance in more complex, cluttered real-world environments.

Summary updated

BibTeX
@inproceedings{scheutz2004robotic,
  author = {Matthias Scheutz and Virgil Andronache and Kathleen M. Eberhard},
  title = {{A Robotic Model of Human Reference Resolution}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2004},
}
Figure