Combining Non-Expert and Expert Crowd Work to Convert Web APIs to Dialog Systems

Ting-Hao K. Huang, Walter S. Lasecki, Alan L. Ritter, Jeffrey P. Bigham

Introduction

Building robust dialog systems, such as chatbots for restaurant recommendations or weather updates, is traditionally a labor-intensive process. Developers must manually map technical API parameters (e.g., lat_long or category_id) to the diverse ways humans communicate. This typically requires extensive domain-specific training data and significant programming effort to handle varied user queries.

This research introduces a hybrid crowdsourcing framework designed to bridge the gap between rigid technical interfaces and fluid human conversation. The goal is to create a "scaffolding" that allows any web API to be converted into a functional conversational agent quickly and cost-effectively.

Methodology

The framework employs a two-phase workflow utilizing both non-expert and expert crowd workers.

The Off-line Phase: Preparation

Before the system goes live, it must establish the "rules" of the API:

  • Question & Answer Collection: Non-expert workers (via Amazon Mechanical Turk) generate natural questions to fulfill specific tasks (e.g., "Where are you located?").
  • Parameter Filtering & Mapping: Workers filter out technical parameters that are unsuitable for conversation and map the remaining ones to the collected questions.
  • Expert Wrapping: Expert developers (recruited from platforms like Upwork) write a minimal Python script—an API Handler—to manage the technical communication between the system and the API.

The On-line Phase: Live Interaction

Once operational, the system follows a three-stage "slot-filling" process to manage live user interactions.

The three-stage on-line workflow: Parameter Value Elicitation, Extraction, and Response Generation.
  1. Parameter Value Elicitation: The system initiates the conversation by asking a question based on the off-line mapping (e.g., "Where are you?").
  2. Parameter Value Extraction: When the user responds, non-expert crowd workers are recruited in real-time to identify and extract specific values (e.g., extracting "Pittsburgh" from "I'm in HCOMP... It's in Pittsburgh").
  3. Response Generation: After the API returns technical data (JSON/XML), crowd workers translate this raw output into a natural, friendly sentence for the user.

Evaluation and Results

The framework was tested using the Yelp restaurant recommendation API. The researchers collected 40 Question-Answer pairs for the task of finding a restaurant. Starting with 22 API parameters, the crowd filtered these down to 9 conversational candidates, ultimately focusing on category_filter, term, and location.

Key Findings

  • Feasibility and Accuracy: The authors successfully built a working dialog system for Yelp. Approximately 65% of the crowd-generated responses correctly mentioned at least one of the top two restaurants returned by the API.
  • Cost-Efficiency: The expert component proved highly affordable; a developer was hired for $15.00 USD and delivered the necessary API wrapper within 12 hours.
  • Strategic Division of Labor: A key insight is the split between "unskilled" and "expert" labor. While experts are needed for the initial API wrapper, non-experts can effectively handle semantic extraction and natural language generation.

Discussion and Future Work

The system demonstrates that human-in-the-loop processing can maintain a conversation by using queuing models to minimize latency. However, relying on crowd workers for every turn introduces more delay than fully automated systems.

Future work aims to address several limitations:

  • Intent Recognition: Incorporating mechanisms to handle users who change their minds or have complex, multi-part goals.
  • Lazy Loading: Learning which parameters are most important over time to reduce the need for the off-line preparation phase.
  • Transition to Automation: Using the crowd-collected data to train machine learning models, eventually phasing out human workers as the system's confidence grows.

Summary updated

Abstract

Thousands of web APIs expose data and services that would be useful to access with natural dialog, from weather and sports to Twitter and movies. The process of adapting each API to a robust dialog system is difficult and time-consuming, as it requires not only programming but also anticipating what is mostly likely to be asked and how it is likely to be asked. We present a crowd-powered system able to generate a natural languageinterface for arbitrary web APIs from scratch without domain-dependent training data or knowledge.Our approach combines two types of crowd workers: non-expert Mechanical Turk workers interpret the functions of the API and elicit information from the user, and expert oDesk workers provide a minimal sufficient scaffolding around the API to allow us to make general queries.We describe our multi-stage process and present results for each stage.
BibTeX
@inproceedings{huang2014combining,
  author = {Ting-Hao K. Huang and Walter S. Lasecki and Alan L. Ritter and Jeffrey P. Bigham},
  title = {{Combining Non-Expert and Expert Crowd Work to Convert Web APIs to Dialog Systems}},
  booktitle = {Hcomp},
  year = {2014},
  doi = {10.1609/hcomp.v2i1.13200},
}
Figure