Introduction
Conversational assistants like Siri or Alexa are increasingly common but remain notoriously difficult and expensive to build. Developers typically must manually create conversational templates for every new domain, which are often brittle and fail to handle natural language variations like slang or temporal references. This creates a scalability gap, leaving a "long tail" of information locked behind thousands of Web APIs (e.g., Yelp, Rotten Tomatoes) inaccessible to current dialog systems.
Guardian addresses this challenge by introducing a hybrid framework that uses on-demand crowd workers to bridge the gap between messy natural language and rigid API structures. By replacing expensive engineering with human intelligence, Guardian can rapidly transform any Web API into a functional spoken dialog system.
Methodology
Guardian operates in two distinct phases: an offline preparation phase and an online execution phase.
Offline Phase: Preparation
Before user interaction, the crowd prepares the API for conversation through three main steps:
- QA Pair Collection: Workers generate potential questions users might ask to retrieve specific information.
- Parameter Filtering: Humans filter out "unnatural" computer-centric parameters (like latitude/longitude) in favor of human-friendly ones (like "city name").
- QA-Parameter Matching: Workers rank questions by importance to ensure the system asks the most helpful follow-up questions first.
Online Phase: Execution
During live interaction, Guardian employs a "Retainer Model"—a pool of pre-recruited workers—to minimize latency.
- Parameter Filling: To ensure accuracy, multiple workers propose values for parameters (e.g., "Chinese food") using an output agreement mechanism.
- Dialog Management: The system tracks the conversation state. If required information is missing, the crowd triggers pre-defined follow-up questions.
- Response Generation: Once the API returns data (often as complex JSON), workers summarize it into natural-sounding sentences for the user.
Evaluation and Results
The researchers evaluated Guardian using eight popular Web APIs, including Yelp, Rotten Tomatoes, and Weather Underground. Evaluation involved workers from CrowdFlower for offline tasks and Amazon Mechanical Turk for real-time online tasks.
Key results include:
- High Accuracy: Guardian achieved a 97% task completion rate across three different domains.
- Low Latency: The retainer model allowed the system to recall 75% of workers within 3 seconds, enabling near-real-time conversation.
- Rapid Deployment: The system could be ported to entirely new APIs in less than a single day.
- Resilience: Unlike traditional systems, the crowd can "recover" from API errors by explaining the situation or suggesting alternatives to the user.
Discussion and Future Work
Guardian demonstrates that "human-in-the-loop" systems can serve as a powerful bootstrap. By collecting logs of human-handled queries, developers can eventually train machine learning models to automate these interactions. A key insight is that pruning APIs for "naturalness" significantly reduces dialog management complexity.
However, limitations remain. While 3-5 seconds is fast for humans, it is slow compared to fully automated systems. Additionally, the cost of human labor makes Guardian better suited for prototyping or high-value complex tasks. Future work will focus on automating the crowd's role and implementing "Intent Recognition" to automatically select the appropriate API from thousands of options based on user requests.
Summary updated