Grounding Natural Language References to Unvisited and Hypothetical Locations

Thomas Emrys Williams, Rehj Cantrell, Gordon Briggs, Paul W. Schermerhorn, Matthias ScheutzAAAI 2013 · DOI

Abstract

Traditional robotic navigation systems often struggle when humans refer to locations not yet present in the robot's internal map. This paper introduces SPEX (SPatial EXpert), a component integrated into the DIARC cognitive architecture that enables robots to ground natural language references to unvisited and hypothetical locations. By treating linguistic descriptions as "hypothetical" entries in a hierarchical map, SPEX allows a robot to learn about its environment through dialogue and successfully navigate to rooms it has never physically encountered. Evaluation in simulated environments shows that the system achieves 100% accuracy in resolving references to both known and previously unknown locations.

Introduction

Most autonomous robots operate under a "static map" assumption, where spatial references like "Go to the break room" can only be resolved if the location is already mapped. This limitation stems from three primary issues identified by the authors: the Discovery Gap, where robots cannot communicate about undiscovered places; the Static Map Constraint, which prevents learning new spatial facts during navigation; and the Action-Coupling Problem, where language is parsed into rigid action sequences rather than persistent spatial knowledge.

To address these, the authors propose a system where natural language is used not just for commands, but as a source of spatial information to build and update the robot's internal world model.

Methodology: The SPEX Component

The core contribution is SPEX, which manages a hierarchical map stratified into a topological layer ($M_{top}$) for rooms and hallways, and a metric layer ($M_{bot}$) for specific coordinates and landmarks.

The DIARC architecture integrated with the SPEX component for spatial reasoning.

As shown in Figure 1, SPEX interacts with several modules within the DIARC architecture:

  • Natural Language Processing (NLP): Provides semantic predicates from speech.
  • Dialogue: Manages clarification requests and context.
  • Goal Manager: Translates spatial references into actionable goals.

Key Algorithms

  1. detectLandmark: Uses sensor data (e.g., Laser Range Finder) to identify landmarks. If a new door is detected, SPEX infers a room exists behind it.
  2. processSemantics: If a description (e.g., "the room at the end of the hall") does not match known locations, SPEX creates a hypothetical entry in the map.
  3. getScript: Generates navigation plans. If coordinates are unknown, it uses linguistic "clues" (e.g., "turn right") to build exploratory sequences.

Experimental Evaluation

The system was evaluated in a simulated office environment featuring hallways and multiple rooms.

A simulated environment used to test navigation to unvisited locations (e.g., from point A to unvisited point B).

The researchers conducted 64 resolution tests covering various descriptive styles. In a "Partial Map" scenario, 44% of large-scale locations were removed from the robot's initial knowledge to test its ability to handle unknown destinations.

  • Full Map Accuracy: 100% (64/64) success.
  • Unknown Location Success: 100% (34/34) success in the partial map evaluation.

The system demonstrated a robust ability to handle ambiguity. For instance, if a room could be at either end of a hallway, SPEX generates an "alternate plan" to check the second location if the first search fails.

Discussion and Key Insights

The success of SPEX highlights several critical concepts in human-robot interaction:

  • Dialogue as a Mapping Tool: Language allows the robot to "know" a room exists before seeing it.
  • Hypothetical Grounding: Placeholder nodes in the topological map allow reasoning about relationships between unknown places (e.g., "Room A is next to Room B").
  • Consolidation: Upon reaching a destination, the robot merges real-time sensor data with its hypothetical placeholders, turning "hearsay" into verified knowledge.

Limitations and Future Work

While successful, the current implementation relies on perfect sensor data and specific linguistic heuristics. Future work aims to incorporate "common sense" spatial reasoning (e.g., where cafeterias are typically located) and more robust perceptual monitoring to handle real-world sensor noise and complex, multi-level building layouts.

Summary updated

BibTeX
@inproceedings{williams2013grounding,
  author = {Thomas Emrys Williams and Rehj Cantrell and Gordon Briggs and Paul W. Schermerhorn and Matthias Scheutz},
  title = {{Grounding Natural Language References to Unvisited and Hypothetical Locations}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2013},
  doi = {10.1609/aaai.v27i1.8563},
}
Figure