Guardian: A Crowd-Powered Spoken Dialog System for Web APIs

Ting-Hao (Kenneth) Huang, Walter S. Lasecki, Jeffrey P. Bigham

Introduction

Conversational assistants like Siri or Alexa are increasingly common but remain notoriously difficult and expensive to build. Developers typically must manually create conversational templates for every new domain, which are often brittle and fail to handle natural language variations like slang or temporal references. This creates a scalability gap, leaving a "long tail" of information locked behind thousands of Web APIs (e.g., Yelp, Rotten Tomatoes) inaccessible to current dialog systems.

Guardian addresses this challenge by introducing a hybrid framework that uses on-demand crowd workers to bridge the gap between messy natural language and rigid API structures. By replacing expensive engineering with human intelligence, Guardian can rapidly transform any Web API into a functional spoken dialog system.

Methodology

Guardian operates in two distinct phases: an offline preparation phase and an online execution phase.

Offline Phase: Preparation

Before user interaction, the crowd prepares the API for conversation through three main steps:

  1. QA Pair Collection: Workers generate potential questions users might ask to retrieve specific information.
  2. Parameter Filtering: Humans filter out "unnatural" computer-centric parameters (like latitude/longitude) in favor of human-friendly ones (like "city name").
  3. QA-Parameter Matching: Workers rank questions by importance to ensure the system asks the most helpful follow-up questions first.
The offline preparation phase involves QA collection, parameter filtering, and mapping.

Online Phase: Execution

During live interaction, Guardian employs a "Retainer Model"—a pool of pre-recruited workers—to minimize latency.

  • Parameter Filling: To ensure accuracy, multiple workers propose values for parameters (e.g., "Chinese food") using an output agreement mechanism.
  • Dialog Management: The system tracks the conversation state. If required information is missing, the crowd triggers pre-defined follow-up questions.
  • Response Generation: Once the API returns data (often as complex JSON), workers summarize it into natural-sounding sentences for the user.
The online execution phase extracts parameters from user input and generates natural language responses.

Evaluation and Results

The researchers evaluated Guardian using eight popular Web APIs, including Yelp, Rotten Tomatoes, and Weather Underground. Evaluation involved workers from CrowdFlower for offline tasks and Amazon Mechanical Turk for real-time online tasks.

Key results include:

  • High Accuracy: Guardian achieved a 97% task completion rate across three different domains.
  • Low Latency: The retainer model allowed the system to recall 75% of workers within 3 seconds, enabling near-real-time conversation.
  • Rapid Deployment: The system could be ported to entirely new APIs in less than a single day.
  • Resilience: Unlike traditional systems, the crowd can "recover" from API errors by explaining the situation or suggesting alternatives to the user.

Discussion and Future Work

Guardian demonstrates that "human-in-the-loop" systems can serve as a powerful bootstrap. By collecting logs of human-handled queries, developers can eventually train machine learning models to automate these interactions. A key insight is that pruning APIs for "naturalness" significantly reduces dialog management complexity.

However, limitations remain. While 3-5 seconds is fast for humans, it is slow compared to fully automated systems. Additionally, the cost of human labor makes Guardian better suited for prototyping or high-value complex tasks. Future work will focus on automating the crowd's role and implementing "Intent Recognition" to automatically select the appropriate API from thousands of options based on user requests.

Summary updated

Abstract

Natural language dialog is an important and intuitive way for people to access information and services. However, current dialog systems are limited in scope, brittle to the richness of natural language, and expensive to produce. This paper introduces Guardian, a crowd-powered framework that wraps existing Web APIs into immediately usable spoken dialog systems. Guardian takes as input the Web API and desired task, and the crowd determines the parameters necessary to complete it, how to ask for them, and interprets the responses from the API. The system is structured so that, over time, it can learn to take over for the crowd. This hybrid systems approach will help make dialog systems both more general and more robust going forward.
BibTeX
@inproceedings{huang2015guardian,
  author = {Ting-Hao (Kenneth) Huang and Walter S. Lasecki and Jeffrey P. Bigham},
  title = {{Guardian: A Crowd-Powered Spoken Dialog System for Web APIs}},
  booktitle = {Hcomp},
  year = {2015},
  doi = {10.1609/hcomp.v3i1.13237},
}
Figure