Introduction
Building robust dialog systems, such as chatbots for restaurant recommendations or weather updates, is traditionally a labor-intensive process. Developers must manually map technical API parameters (e.g., lat_long or category_id) to the diverse ways humans communicate. This typically requires extensive domain-specific training data and significant programming effort to handle varied user queries.
This research introduces a hybrid crowdsourcing framework designed to bridge the gap between rigid technical interfaces and fluid human conversation. The goal is to create a "scaffolding" that allows any web API to be converted into a functional conversational agent quickly and cost-effectively.
Methodology
The framework employs a two-phase workflow utilizing both non-expert and expert crowd workers.
The Off-line Phase: Preparation
Before the system goes live, it must establish the "rules" of the API:
- Question & Answer Collection: Non-expert workers (via Amazon Mechanical Turk) generate natural questions to fulfill specific tasks (e.g., "Where are you located?").
- Parameter Filtering & Mapping: Workers filter out technical parameters that are unsuitable for conversation and map the remaining ones to the collected questions.
- Expert Wrapping: Expert developers (recruited from platforms like Upwork) write a minimal Python script—an API Handler—to manage the technical communication between the system and the API.
The On-line Phase: Live Interaction
Once operational, the system follows a three-stage "slot-filling" process to manage live user interactions.
- Parameter Value Elicitation: The system initiates the conversation by asking a question based on the off-line mapping (e.g., "Where are you?").
- Parameter Value Extraction: When the user responds, non-expert crowd workers are recruited in real-time to identify and extract specific values (e.g., extracting "Pittsburgh" from "I'm in HCOMP... It's in Pittsburgh").
- Response Generation: After the API returns technical data (JSON/XML), crowd workers translate this raw output into a natural, friendly sentence for the user.
Evaluation and Results
The framework was tested using the Yelp restaurant recommendation API. The researchers collected 40 Question-Answer pairs for the task of finding a restaurant. Starting with 22 API parameters, the crowd filtered these down to 9 conversational candidates, ultimately focusing on category_filter, term, and location.
Key Findings
- Feasibility and Accuracy: The authors successfully built a working dialog system for Yelp. Approximately 65% of the crowd-generated responses correctly mentioned at least one of the top two restaurants returned by the API.
- Cost-Efficiency: The expert component proved highly affordable; a developer was hired for $15.00 USD and delivered the necessary API wrapper within 12 hours.
- Strategic Division of Labor: A key insight is the split between "unskilled" and "expert" labor. While experts are needed for the initial API wrapper, non-experts can effectively handle semantic extraction and natural language generation.
Discussion and Future Work
The system demonstrates that human-in-the-loop processing can maintain a conversation by using queuing models to minimize latency. However, relying on crowd workers for every turn introduces more delay than fully automated systems.
Future work aims to address several limitations:
- Intent Recognition: Incorporating mechanisms to handle users who change their minds or have complex, multi-part goals.
- Lazy Loading: Learning which parameters are most important over time to reduce the need for the off-line preparation phase.
- Transition to Automation: Using the crowd-collected data to train machine learning models, eventually phasing out human workers as the system's confidence grows.
Summary updated