Demonstration of PLOW: A Dialogue System for One-Shot Task Learning

James Allen, Nathanael Chambers, George Ferguson, Lucian Galescu, Hyuckchul Jung, Mary Swift, William TaysomNAACL 2007

TL;DR

This paper introduces PLOW, a system that learns complex web-based tasks from a single demonstration combined with natural language narration. Unlike previous methods requiring multiple examples, PLOW leverages "play-by-play" descriptions to infer higher-level task structures, semantic relationships, and control constructs, enabling more robust and adaptable task learning.

What Problem Does This Paper Address?

Learning new tasks, especially complex ones involving sequences of actions and decision-making, is challenging for AI systems. Traditional approaches often require numerous demonstrations to generalize effectively, which is impractical for users. Furthermore, simply observing actions doesn't reveal the underlying intent, control flow, or semantic meaning behind those actions. This paper addresses the problem of "one-shot task learning," where a system must learn a new procedure from a single demonstration, by integrating visual actions with natural language explanations. This allows the system to go beyond rote memorization and understand the "why" behind the actions, leading to more flexible and reliable task execution.

Methods

PLOW employs a mixed-initiative approach where a user demonstrates a task on a web browser while providing a natural language "play-by-play" commentary. The system combines this linguistic input with the observed actions to build a parameterized procedure. Key to PLOW's method is the use of natural language to infer:

  • Useful levels of abstraction: Understanding what constitutes a meaningful step in a task.
  • Parameter dependencies: Identifying how different parts of an action relate to each other.
  • Hierarchical structure: Recognizing sub-tasks and their organization.
  • Semantic relationships: Connecting linguistic descriptions of objects with their actual representations in the environment.
  • Control constructs: Inferring loops, conditions, and other logic not directly visible in the demonstration.

Specialized reasoning modules within PLOW collaborate to interpret user intentions, build the task model, and ensure consistency with existing knowledge. The system also learns to identify and unify objects mentioned in the NL with their semantic representations, allowing for reliable object recognition in dynamic environments.

Datasets

Not discussed in the paper. The evaluation involved human subjects learning tasks using PLOW and other systems, with performance scored based on a complex predefined criteria.

Key Insights and Ideas

  • Natural Language as a Bridge: The "play-by-play" narration is crucial for bridging the gap between observed actions and understanding the underlying task logic and semantics. It provides the missing context that a purely observational approach lacks.
  • One-Shot Learning is Achievable: PLOW demonstrates that learning complex procedures from a single demonstration is feasible when augmented with rich linguistic feedback.
  • Robust Object Identification: By unifying linguistic descriptions with semantic representations, PLOW can reliably find objects in complex and dynamic web environments, a significant challenge in many agent-based systems.
  • Mixed-Initiative Learning: The system's ability to collaborate with the user, asking clarifying questions (e.g., "When should I stop searching?") and executing actions, makes the learning process more efficient and interactive.

Key Results

In an evaluation involving 16 human subjects and four task learning systems, PLOW achieved the highest average score on learned tasks and was preferred by users more than half the time. While specific quantitative results for task learning accuracy are not detailed, the user preference and high average score indicate strong performance and user satisfaction compared to other systems.

Limitations and Future Work

The paper acknowledges that the system's ability to handle complex control structures and infer deeper semantic relationships is an ongoing area of research. Future work would likely involve expanding the range of tasks PLOW can learn, improving its ability to handle more ambiguous or underspecified demonstrations, and further refining its natural language understanding capabilities. The authors also suggest demonstrating PLOW live and making it available for extended use during conferences, indicating a desire for broader adoption and feedback.

Summary updated

BibTeX
@inproceedings{allen2007demonstration,
  author = {James Allen and Nathanael Chambers and George Ferguson and Lucian Galescu and Hyuckchul Jung and Mary Swift and William Taysom},
  title = {{Demonstration of PLOW: A Dialogue System for One-Shot Task Learning}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2007},
}
Figure