Sorry and I Didn't Catch That! - An Investigation of Non-understanding Errors and Recovery Strategies

Dan Bohus, Alexander I. RudnickySIGDIAL Workshop 2005 · OpenReview

TL;DR

This paper investigates "non-understanding" errors in spoken dialog systems, where the system fails to interpret user input. It analyzes the causes and impact of these errors, compares ten different recovery strategies, and demonstrates that a more informed recovery policy significantly improves performance, especially for non-native speakers.

What Problem Does This Paper Address?

Spoken dialog systems (SDSs) are notoriously brittle, often failing to understand user input due to the inherent unreliability of speech recognition. This leads to "non-understanding" errors, where the system simply doesn't get what the user said. These errors can severely degrade the user experience and task completion rates, limiting the naturalness and effectiveness of SDSs. While systems know when a non-understanding occurs, they often lack a robust mechanism to diagnose the cause or select the best strategy to recover. This paper aims to systematically analyze these errors, evaluate various recovery strategies, and explore how to improve the system's ability to handle them effectively.

Methods

The researchers conducted a user study using a conference room reservation system called RoomLine. They employed a mixed-initiative approach, allowing both the user and system to guide the conversation.

  • Data Collection: 46 participants interacted with RoomLine. They used two conditions: a "control" group where the system randomly chose one of ten recovery strategies after a non-understanding, and a "wizard" group where a human operator (the wizard) manually selected the best recovery strategy, having access to the live user speech and system state.
  • Error Analysis: Dialogs were transcribed and annotated to identify sources of understanding errors (e.g., ASR errors, out-of-grammar, out-of-application scope) and the types of user responses following non-understandings.
  • Strategy Evaluation: The performance of ten distinct recovery strategies (e.g., asking to repeat, rephrase, providing more help, moving on to a different task) was evaluated based on their recovery rate (the proportion of successful recoveries).
  • Policy Evaluation: The performance of the wizard's informed policy was compared against the control group's uninformed policy to assess the impact of a better recovery strategy selection.
  • Predictive Modeling: Logistic regression models were used to predict the likelihood of success for each recovery strategy based on runtime features.

Datasets

The study utilized a newly collected corpus of dialogs from 46 participants interacting with the RoomLine conference room reservation system. This corpus, comprising 449 sessions and 8278 user turns, was annotated for various error types, user responses, and task success.

Key Insights and Ideas

  • ASR is the primary culprit, but not the only one: While speech recognition (ASR) errors are the largest source of non-understandings, a significant portion (~30%) stems from users requesting features outside the system's scope or using out-of-grammar expressions. Users are often unaware of the system's full capabilities.
  • "MoveOn" is surprisingly effective: The strategy of abandoning the current problematic question and moving to a different part of the task (MoveOn) or providing comprehensive help (Help, TerseYouCanSay) proved to be the most successful recovery strategies when applied without prior knowledge. This suggests that sometimes, it's better to pivot than to fix.
  • Informed policies matter, especially for struggling users: A human wizard's informed strategy selection significantly improved task success and other performance metrics, particularly for non-native speakers who initially struggled more with the system. This highlights the potential for smarter, context-aware recovery.
  • User response types are influenced by strategy: Different recovery strategies elicit different user responses. For example, the MoveOn strategy led to more "change" responses, while AskRepeat led to more "repeat" responses.
  • Learning recovery policies is feasible: Preliminary results suggest that it's possible to build predictive models for strategy success based on runtime features, paving the way for learning optimal recovery policies from data.

Key Results

  • Error Sources: ASR errors accounted for 62% of non-understandings, with out-of-application and out-of-grammar errors making up a substantial portion of the remainder.
  • Strategy Performance (Uninformed): The top-performing strategies were MoveOn (64.4% recovery rate), Help (58.5%), and TerseYouCanSay (56.5%). The least effective were Yield (31.2%) and AskRepeat (33.7%).
  • Impact of Informed Policy: The wizard policy led to significant improvements in task success, especially for non-native users (task success rose from 31.6% to 57.4%). Recovery rates also improved.
  • User Response Impact: "Change" responses had the highest recovery rate (63%), followed by "repeat" and "rephrase" responses.

Limitations and Future Work

  • Wizard's Advantage: The wizard had access to live speech, a privilege not available to an automated system. Therefore, demonstrating a performance gap doesn't prove an automated system can achieve it, only that a better policy exists.
  • Generalizability: The authors caution that the success of specific strategies and user responses may not generalize across different domains or tasks.
  • Learning Policy Challenges: Learning optimal recovery policies is complicated by the small number of instances per strategy and the risk of overfitting.
  • Future Directions: The paper suggests exploring more sophisticated "you-can-say" help messages, preemptively offering help to "problematic" user groups, and developing more appropriate metrics for evaluating recovery performance. Further work is planned to develop and test learned recovery policies in a live system.

Summary updated

BibTeX
@inproceedings{bohus2005sorry,
  author = {Dan Bohus and Alexander I. Rudnicky},
  title = {{Sorry and I Didn't Catch That! - An Investigation of Non-understanding Errors and Recovery Strategies}},
  booktitle = {SIGDIAL Workshop},
  year = {2005},
  url = {https://openreview.net/forum?id=Kn56R3F317G},
}
Figure