TL;DR
This paper introduces an "embedded Wizard-of-Oz" (WOz) methodology to study how humans handle uncertainty in voice search for spoken dialogue systems. They found that the best human "wizards" don't just blindly offer the top search result but can effectively determine when the desired item isn't present, a skill that can be learned and modeled using system features.
What Problem Does This Paper Address?
Spoken dialogue systems (like those used for booking flights or finding restaurants) often struggle with noisy Automatic Speech Recognition (ASR) output, leading to misunderstandings. When a system can't confidently interpret a user's request, it often resorts to asking the user to repeat themselves, which can be frustrating and degrade performance. This paper tackles the challenge of improving understanding, particularly for complex information retrieval tasks like searching a large book database by title, where ASR errors can easily lead to misinterpretations. They propose "voice search" as a mechanism to mitigate these issues, where the system queries a database with the ASR hypothesis to find potential matches, but the core problem remains: how does a human (or a system emulating a human) effectively use these search results, especially when the ASR is imperfect?
Methods
The authors developed an "embedded Wizard-of-Oz" (WOz) setup. In this paradigm, a human "wizard" acts as the system's backend, interpreting ASR output and making decisions. The key innovation here is that the wizard is presented with the ASR hypothesis and the results of a voice search query against a real database. This allows them to observe how wizards handle uncertainty when the ASR might be wrong, but the voice search provides potential candidates.
They collected data by having participants act as both users and wizards in a book-finding task. The wizard's interface displayed ASR hypotheses and ranked search results. The wizard's task was to choose the correct title, ask a clarifying question, or indicate that the title wasn't found.
To model wizard behavior, they engineered 60 features, including information visible on the wizard's GUI (like search result ranking and confidence scores) and features not directly visible (like ASR confidence and acoustic model scores). They then used machine learning techniques (decision trees, logistic regression, and linear regression) to predict wizard actions based on these features.
Datasets
The study used a simulated library database containing 71,166 book titles and 28,031 authors. A trigram language model was built from a subset of 7,500 book titles. The experiment itself generated a dataset of 4172 "title cycles" (interactions where a user requests a book title). No new benchmark datasets were introduced, but the collected data serves as a valuable resource for studying human-system interaction in the context of noisy ASR and voice search.
Key Insights and Ideas
- Voice Search as a Mitigation Strategy: Voice search, by querying a database with the ASR hypothesis, can help resolve ambiguities and find correct matches even with imperfect ASR.
- The Importance of "Not Offering": The most successful wizard (W4) excelled not just at finding correct titles but, crucially, at recognizing when the requested title was not present in the search results. This ability to "not offer" a candidate was key to her high accuracy.
- Modeling Human Uncertainty: Human wizards' decisions can be effectively modeled using a combination of features directly visible to them (like search result ranking) and internal system metrics (like ASR confidence scores). This suggests that dialogue systems can learn to emulate human-like uncertainty handling.
- Feature Importance: Features like the "DisplayType" of the search return (e.g., singleton result vs. a list), "RecentSuccess" (how often the wizard chose correctly recently), and "ContiguousWordMatch" (exact word overlap between ASR and candidates) were important in predicting wizard actions.
Key Results
- High Accuracy for Correct Titles: Wizards were generally good at identifying the correct title when it was present in the search results (baseline accuracy of ~65% by always offering the first return).
- Wizard Performance Variation: There was significant variation in wizard performance, particularly in their ability to correctly identify when a title was not present. Wizard W4 achieved the highest accuracy by being more conservative in offering titles.
- Machine Learning Model Performance: Decision tree models achieved 82.2% accuracy in predicting wizard actions, outperforming logistic regression (75.2%). These models could predict confident choices well but were less consistent with tentative choices or questions.
- Feature-Based Modeling: The learned models for the best wizard (W4) combined features like search result presentation, recent success, and ASR confidence scores, demonstrating the value of integrating system-level information.
Limitations and Future Work
- Limited Task Scope: The study focused on a single turn exchange (book title requests) rather than full dialogues. This limits generalizability to more complex conversational scenarios.
- Simulated User: The "users" were instructed to read synopses and reorder titles, which may not fully replicate natural user behavior.
- Wizard Expertise: The best wizard (W4) demonstrated a level of nuanced decision-making that might be difficult to fully replicate in a fully automated system.
- Future Work: The authors plan to integrate voice search into their CheckItOut dialogue system and conduct further experiments with embedded wizards in full dialogues. They aim to model the most successful wizards and explore more sophisticated NLU approaches, including statistical machine learning for database entity recognition. They also plan to create datasets biased towards cases where "not offering" is the correct action to better train systems in this crucial skill.
Summary updated