Learning Web-based Procedures by Reasoning over Explanations and Demonstrations in Context

Shashank Srivastava, Oleksandr Polozov, Nebojsa Jojic, Christopher Meek

Abstract

We explore learning web-based tasks from a human teacher through natural language explanations and a single demonstration. Our approach investigates a new direction for semantic parsing that models explaining a demonstration in a context, rather than mapping explanations to demonstrations. By leveraging the idea of inverse semantics from program synthesis to reason backwards from observed demonstrations, we ensure that all considered interpretations are consistent with executable actions in any context, thus simplifying the problem of search over logical forms. We present a dataset of explanations paired with demonstrations for web-based tasks. Our methods show better task completion rates than a supervised semantic parsing baseline (40% relative improvement on average), and are competitive with simple exploration-and-demonstration based methods, while requiring no exploration of the environment. In learning to align explanations with demonstrations, basic properties of natural language syntax emerge as learned behavior. This is an interesting example of pragmatic language acquisition without any linguistic annotation.

Introduction

Automating repetitive web tasks—such as booking flights, forwarding specific emails, or managing calendars—remains a significant goal for personal AI assistants. However, current systems are largely limited to pre-programmed commands. Teaching an AI a new procedure typically requires either massive supervised datasets of language-to-code pairs or extensive reinforcement learning (RL) where an agent explores an environment through trial and error. The latter is often impractical in real-world web environments where erroneous actions can have permanent consequences.

This paper introduces LED (Learning from Explained Demonstrations), a framework designed to tackle the "one-shot" challenge: enabling an AI to learn a procedure from a single demonstration paired with brief natural language explanations, much like a human intern would.

Methodology: Inverse Semantics

The core innovation of LED is a paradigm shift in semantic parsing. Traditional parsers attempt to map a sentence directly to a logical form (code). LED flips this process using Inverse Semantics, a concept borrowed from program synthesis.

Instead of searching the entire space of possible code, the agent first observes a demonstration. It then identifies all logical forms within a Domain Specific Language (DSL) that would actually result in the observed action within that specific web context. For example, if a user clicks a blue button, the agent only considers logical forms consistent with that click, immediately discarding irrelevant actions like typing text. Finally, the agent uses a generative model to find which of these consistent logical forms best aligns with the user's natural language explanation.

The LED framework reasons backwards from a demonstration to find consistent logical forms that align with the explanation in context.

The DSL describes web elements based on their HTML tags, text content, and spatial relationships (e.g., "to the right of" or "at the top"). This grounding ensures that the language is always interpreted in the physical context of the webpage.

Dataset and Experimental Setup

The study utilizes the Mini World-of-Bits (MiniWoB) framework, a collection of sandbox web tasks. The authors collected a new benchmark dataset consisting of 520 demonstrations across 40 different tasks. Each demonstration is paired with stepwise natural language instructions provided by human workers, totaling 1,719 steps. While the vocabulary is relatively small (995 words), the language is diverse and includes complex spatial reasoning and references to specific strings on the page.

Key Results

LED demonstrates significant performance gains and data efficiency compared to existing baselines. The model showed a 40% relative improvement in task completion rates over a standard supervised semantic parsing baseline.

Task completion rates across various web tasks comparing LED to supervised semantic parsing and reinforcement learning baselines.

Notably, LED achieved results competitive with state-of-the-art reinforcement learning methods (BC+RL), despite the RL methods requiring 200 times more demonstrations and extensive environment exploration. In predicting the correct action from an individual explanation, the LED (+Syntax) variant reached 45% accuracy, significantly outperforming the supervised baseline (39%) and random choice (28%).

Discussion and Insights

  • Pragmatics over Syntax: By rooting language in the physical context of the webpage and the demonstration, the model does not require prior grammatical knowledge. It learns that the word "click" relates to a specific operator because that is the action occurring in the demonstration.
  • Search Space Reduction: Traditional parsers often struggle with the "search explosion" of possible code. LED bypasses this by using the demonstration as a hard constraint—if the code doesn't match the demo, it's discarded immediately.
  • Emergent Behavior: The model began to learn basic linguistic patterns, such as Verb-Subject-Object order, entirely from scratch by attempting to align words with the features of the elements being interacted with.

Limitations and Future Work

Despite its success, the model currently struggles with discourse referents (e.g., "it" or "the previous one"), treating most steps as independent events. Furthermore, human teachers often skip "obvious" steps that the AI currently requires for perfect alignment. Future work aims to handle complex conditional logic (e.g., "If the price is under $20, click buy") and incorporate interactivity, allowing the agent to ask clarifying questions when explanations are ambiguous.

Summary updated

Abstract

We explore learning web-based tasks from a human teacher through natural language explanations and a single demonstration. Our approach investigates a new direction for semantic parsing that models explaining a demonstration in a context, rather than mapping explanations to demonstrations. By leveraging the idea of inverse semantics from program synthesis to reason backwards from observed demonstrations, we ensure that all considered interpretations are consistent with executable actions in any context, thus simplifying the problem of search over logical forms. We present a dataset of explanations paired with demonstrations for web-based tasks. Our methods show better task completion rates than a supervised semantic parsing baseline (40% relative improvement on average), and are competitive with simple exploration-and-demonstration based methods, while requiring no exploration of the environment. In learning to align explanations with demonstrations, basic properties of natural language syntax emerge as learned behavior. This is an interesting example of pragmatic language acquisition without any linguistic annotation.
BibTeX
@inproceedings{srivastava2020learning,
  author = {Shashank Srivastava and Oleksandr Polozov and Nebojsa Jojic and Christopher Meek},
  title = {{Learning Web-based Procedures by Reasoning over Explanations and Demonstrations in Context}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2020},
  url = {https://openreview.net/forum?id=VYEJyUR0Qif},
}
Figure