Abstract
This paper presents Rosie, a robotic agent designed for Interactive Task Learning (ITL). Unlike traditional autonomous systems that rely on rigid programming or extensive training datasets, Rosie learns novel tasks through real-time natural language instructions and visual demonstrations. Built on the Soar cognitive architecture, Rosie utilizes an integrated NLP parser and a knowledge compilation mechanism known as "chunking" to transform declarative instructions into efficient procedural rules. We demonstrate Rosie's capabilities across tabletop and mobile domains, showing it can learn and solve complex puzzles and games with efficiency comparable to hand-coded systems.
Introduction
Most contemporary autonomous systems are "brittle," relying on hand-coded logic for specific tasks or requiring massive datasets for reinforcement learning. This creates a significant barrier for non-expert users who wish to teach a robot a new task on the fly. This research addresses the challenge of Interactive Task Learning (ITL), aiming to bridge the gap between human-level instruction and robotic execution. The goal is an agent capable of learning "from scratch"—understanding termination conditions, legal moves, and task-specific concepts—using natural language similar to how a human might teach a peer.
Methods
Rosie is built upon the Soar cognitive architecture, employing a multi-modal and hierarchical approach to learning:
- Integrated NLP Parser: Rosie uses a specialized parser within Soar that integrates semantic and pragmatic knowledge to understand the intent behind instructions, rather than just syntactic structure.
- Knowledge Compilation (Chunking): When first taught a task, Rosie reasons through rules using declarative structures. Soar’s "chunking" mechanism dynamically compiles this knowledge into procedural rules, allowing the robot to transition from deliberate reasoning to rapid execution.
- Visual Demonstration: Through "situated interaction," an instructor can arrange objects and define goal states visually. Rosie then hypothesizes the underlying conditions required to reach that state.
- Concept Transfer: Rosie can reuse learned concepts across tasks. For instance, the logic for "three-in-a-row" learned in Tic-Tac-Toe can be automatically applied to other games like Three Men’s Morris.
Experimental Setup
Rosie was evaluated across several interactive environments rather than static datasets:
- Tabletop Domain: A robot arm paired with a Kinect sensor for manipulating objects.
- Mobile Domain: Both a real-world mobile robot and a high-fidelity simulation for delivery missions.
- Task Suite: The agent was taught rules for various puzzles, including the Eight Puzzle, Tic-Tac-Toe, and an isomorphism of the Tower of Hanoi adapted for robotic dexterity.
Key Insights and Results
One of Rosie's primary strengths is its ability to handle recursive definitions. If an instructor uses an unknown term, the agent proactively asks for a definition, building complex semantic trees from simple primitives. Furthermore, the system utilizes isomorphisms for physicality; for example, teaching Tower of Hanoi using different-sized blocks on flat locations to accommodate the robot's physical constraints while maintaining the game's underlying logic.
Key results include:
- Rapid Problem Solving: Once rules are learned, Rosie can solve puzzles like the Tower of Hanoi in approximately one second.
- Efficiency: The chunking mechanism ensures that Rosie’s reasoning speed is comparable to hand-coded software.
- Versatility: The same cognitive "brain" was successfully deployed in both a stationary tabletop arm and a mobile robot that had to reason about objects outside its immediate field of view.
Discussion and Future Work
While Rosie demonstrates significant progress in ITL, several limitations remain. The NLP parser currently relies on a restricted form of natural language and cannot yet handle the full ambiguity of open-ended human conversation. Additionally, reasoning about objects outside of immediate perception in the mobile domain remains a hurdle. Future work will focus on scaling these capabilities to more complex, dynamic environments and improving the physical dexterity of the robotic platforms to reduce the need for task isomorphisms.
Summary updated