Abstract
This research introduces a robot dialog agent that learns to understand natural language commands through real-time interaction. By leveraging clarification dialogues and user paraphrases as automated training data, the system incrementally improves its semantic parser. This approach significantly reduces user frustration and increases task success rates without the need for massive pre-annotated datasets.
Introduction: The Challenge of Natural Language Interaction
For robots to be effective in everyday environments like homes and offices, they must understand instructions from non-expert users. Traditional approaches often rely on brittle keyword searches or expensive, large-scale supervised learning. Keyword systems fail when users deviate from specific command syntax, while supervised learning requires thousands of hand-annotated examples that do not easily adapt to new environments.
The authors address this "cold start" problem by developing a system that learns linguistic quirks and environmental context through natural conversation, starting with minimal prior knowledge.
System Architecture and Methods
The dialog agent is composed of three integrated components designed to map language to action:
- Semantic Parser: Utilizes Combinatory Categorial Grammar (CCG) and $\lambda$-calculus to translate natural language into executable logical forms.
- Grounding via Knowledge Base: Connects abstract concepts (e.g., "office") to specific real-world entities (e.g., "Room 3508") by querying an environmental map.
- Dialog Manager: Maintains a "belief state"—a probability distribution over the user's intent. If the agent's confidence is low, it initiates clarification questions.
Incremental Learning through Paraphrase
The core innovation is the use of clarification as a supervision signal. When a user provides a paraphrase after a failure (e.g., changing "Report to 3418" to "Go to room 3418"), the system automatically pairs the original failed phrase with the successful logical form. This allows the robot to learn new vocabulary and phrasing on the fly.
Experimental Setup
The system was evaluated using a minimal seed lexicon of 105 entries and 5 supervised pairs. Training data was collected through Mechanical Turk (MTurk) interactions and real-world testing on a physical Segbot in a university office building over four days.
Key Results
The agent demonstrated significant improvement through incremental learning in both simulated and physical environments.
- Efficiency Gains: On MTurk, the average number of dialog turns required for delivery tasks dropped from 17.9 to 3.82 after three training batches.
- User Satisfaction: Survey data indicated a significant increase in users feeling "understood" (rising from 2.21 to 2.84) and a corresponding decrease in "frustration" (dropping from 2.19 to 1.5).
- Real-World Performance: On the physical Segbot, the success rate for delivery tasks improved from 20% to 60% after four days of interaction-based learning.
Discussion and Future Directions
While successful, the current system has several areas for growth. The authors suggest moving from a fixed rule-based dialog policy to a POMDP-based policy to improve robustness under uncertainty. Future work also aims to integrate speech recognition and visual perception, allowing the robot to learn from auditory cues and physical pointing, and to expand the action space beyond navigation and delivery.
Summary updated