Learning to Interpret Natural Language Commands through Human-Robot Dialog

Jesse Thomason, Shiqi Zhang, Raymond J. Mooney, Peter Stone

Abstract

This research introduces a robot dialog agent that learns to understand natural language commands through real-time interaction. By leveraging clarification dialogues and user paraphrases as automated training data, the system incrementally improves its semantic parser. This approach significantly reduces user frustration and increases task success rates without the need for massive pre-annotated datasets.

Introduction: The Challenge of Natural Language Interaction

For robots to be effective in everyday environments like homes and offices, they must understand instructions from non-expert users. Traditional approaches often rely on brittle keyword searches or expensive, large-scale supervised learning. Keyword systems fail when users deviate from specific command syntax, while supervised learning requires thousands of hand-annotated examples that do not easily adapt to new environments.

The authors address this "cold start" problem by developing a system that learns linguistic quirks and environmental context through natural conversation, starting with minimal prior knowledge.

System Architecture and Methods

The dialog agent is composed of three integrated components designed to map language to action:

  1. Semantic Parser: Utilizes Combinatory Categorial Grammar (CCG) and $\lambda$-calculus to translate natural language into executable logical forms.
  2. Grounding via Knowledge Base: Connects abstract concepts (e.g., "office") to specific real-world entities (e.g., "Room 3508") by querying an environmental map.
  3. Dialog Manager: Maintains a "belief state"—a probability distribution over the user's intent. If the agent's confidence is low, it initiates clarification questions.
The dialog agent architecture, showing the flow from user text through semantic parsing and grounding to belief state updates and response generation.

Incremental Learning through Paraphrase

The core innovation is the use of clarification as a supervision signal. When a user provides a paraphrase after a failure (e.g., changing "Report to 3418" to "Go to room 3418"), the system automatically pairs the original failed phrase with the successful logical form. This allows the robot to learn new vocabulary and phrasing on the fly.

Experimental Setup

The system was evaluated using a minimal seed lexicon of 105 entries and 5 supervised pairs. Training data was collected through Mechanical Turk (MTurk) interactions and real-world testing on a physical Segbot in a university office building over four days.

Key Results

The agent demonstrated significant improvement through incremental learning in both simulated and physical environments.

Graphs showing the improvement in survey responses (left) and the reduction in user dialog turns (right) as the number of training batches increases.
  • Efficiency Gains: On MTurk, the average number of dialog turns required for delivery tasks dropped from 17.9 to 3.82 after three training batches.
  • User Satisfaction: Survey data indicated a significant increase in users feeling "understood" (rising from 2.21 to 2.84) and a corresponding decrease in "frustration" (dropping from 2.19 to 1.5).
  • Real-World Performance: On the physical Segbot, the success rate for delivery tasks improved from 20% to 60% after four days of interaction-based learning.

Discussion and Future Directions

While successful, the current system has several areas for growth. The authors suggest moving from a fixed rule-based dialog policy to a POMDP-based policy to improve robustness under uncertainty. Future work also aims to integrate speech recognition and visual perception, allowing the robot to learn from auditory cues and physical pointing, and to expand the action space beyond navigation and delivery.

Summary updated

Abstract

Intelligent robots frequently need to understand requests from naive users through natural language. Previous approaches either cannot account for language variation, e.g., keyword search, or require gathering large annotated corpora, which can be expensive and cannot adapt to new variation. We introduce a dialog agent for mobile robots that understands human instructions through semantic parsing, actively resolves ambiguities using a dialog manager, and incrementally learns from human-robot conversations by inducing training data from user paraphrases. Our dialog agent is implemented and tested both on a web interface with hundreds of users via Mechanical Turk and on a mobile robot over several days, tasked with understanding navigation and delivery requests through natural language in an office environment. In both contexts, We observe significant improvements in user satisfaction after learning from conversations.
BibTeX
@inproceedings{thomason2015learning,
  author = {Jesse Thomason and Shiqi Zhang and Raymond J. Mooney and Peter Stone},
  title = {{Learning to Interpret Natural Language Commands through Human-Robot Dialog}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2015},
}
Figure