Interactive Task Learning from GUI-Grounded Natural Language Instructions and Demonstrations

Toby Jia-Jun Li, Tom Mitchell, Brad A. MyersACL 2020

TL;DR

This paper introduces SUGILITE, an intelligent agent that learns new tasks from users by combining natural language instructions with demonstrations on mobile app GUIs. It addresses the challenge of enabling non-programmers to automate complex, "long-tail" tasks by providing a mixed-initiative system that clarifies intent and generalizes learned concepts.

What Problem Does This Paper Address?

Modern virtual assistants like Siri and Alexa are limited to a pre-programmed set of tasks, failing to meet the diverse needs of users. While developers can create new "skills," this requires significant programming expertise. Interactive Task Learning (ITL) aims to bridge this gap by allowing users to teach agents new tasks through natural interaction. However, existing ITL systems face challenges in usability (requiring programming knowledge), applicability (limited to specific domains), generalizability (difficulty adapting to new contexts), flexibility (limited rule definition), and robustness (brittleness to changes). This paper tackles these challenges by enabling users to teach complex tasks through a combination of spoken instructions and direct demonstrations on mobile app GUIs, making task automation accessible to a broader audience.

Methods

SUGILITE employs a multi-modal, mixed-initiative approach to learn tasks. The core idea is to combine natural language instructions with demonstrations on mobile app GUIs.

  • GUI-Grounded Natural Language: Users provide instructions that are grounded in the visual elements of the app's GUI. This helps the agent understand concepts and parameters by referencing visual cues.
  • Demonstrations: When the agent encounters an unknown concept or procedure, it prompts the user to demonstrate it using the app's GUI. This is a form of "programming by demonstration" (PBD).
  • Mixed-Initiative Interaction: The system and user engage in a dialogue to clarify intent. The agent asks targeted questions (e.g., "How do I tell whether it is hot?") based on the type of information it needs (e.g., a boolean condition or a value).
  • Intent Clarification: For demonstrated actions, SUGILITE uses a multi-turn conversation to help users explain their intent. This addresses the "data description problem" in PBD, where a literal demonstration might not capture the user's true goal. The system represents GUI screens as graphs to parse user explanations and ground them to specific UI elements.
  • Task Parameterization and Generalization: SUGILITE infers task parameters and their possible values by analyzing the GUI structure during demonstrations. This allows it to generalize learned procedures (e.g., "order any drink from Starbucks") and concepts (e.g., "hot" can refer to weather or an oven temperature) to new contexts and domains.

Datasets

Not explicitly discussed in the paper. The system relies on interacting with existing third-party mobile applications and their GUIs.

Key Insights and Ideas

  • GUIs as a Bridge for Non-Programmers: Leveraging the familiar interface of mobile apps significantly lowers the barrier to entry for users wanting to teach automation tasks. GUIs provide a concrete grounding for abstract natural language instructions.
  • Mixed-Initiative Dialogue is Crucial for Ambiguity Resolution: Simply observing a demonstration isn't enough. The system needs to actively engage the user in a dialogue to understand the intent behind the actions, especially when multiple interpretations are possible.
  • Generalization is Key to Reusability: The ability to generalize learned procedures and concepts beyond the specific example demonstrated is what makes the learning truly valuable and saves users from repetitive teaching.
  • "Speak and Point" Interaction is Powerful: Combining speech with direct manipulation (pointing on the GUI) offers a natural and effective way for users to convey information and for the system to disambiguate.

Key Results

The paper focuses on the system's design and capabilities rather than presenting extensive quantitative results. However, the evaluation section highlights:

  • End users without programming expertise were able to successfully teach SUGILITE common tasks, conditional rules, and concepts.
  • Users were able to clarify ambiguities in their demonstrated actions.
  • Most participants found SUGILITE easy and natural to use.
  • Teaching a task took users 3-6 times longer than performing it manually, but this is seen as an investment that saves time for repetitive tasks.

Limitations and Future Work

  • Robustness of Natural Language Understanding: The current grammar-based semantic parser can be rigid. Future work involves exploring neural parsers and leveraging pre-trained language models (like BERT) for more flexible understanding.
  • Extracting Task Semantics from GUIs: While SUGILITE uses GUI grounding, deeper extraction of semantic information from GUIs could further simplify high-level task specifications for users.
  • Combining User-Specific and General Models: Integrating user-specific learned models with pre-trained, domain-agnostic models is a future challenge.
  • Proactive Agent Behavior: Developing agents that are more proactive in guiding users and taking initiative in the dialogue is an area for future exploration.
  • User Privacy: Collecting and aggregating personal task instructions raises user privacy concerns that need to be addressed.

Summary updated

BibTeX
@inproceedings{li2020interactive,
  author = {Toby Jia-Jun Li and Tom Mitchell and Brad A. Myers},
  title = {{Interactive Task Learning from GUI-Grounded Natural Language Instructions and Demonstrations}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2020},
}
Figure