TL;DR
This paper investigates user satisfaction with different error recovery strategies in spoken dialogue systems (SDS). It finds that a modified "MoveOn" strategy, which attempts to guess the user's intent without explicit confirmation, leads to the highest user satisfaction, especially among experienced users. This suggests that prioritizing user experience, not just task success, is crucial for effective SDS design.
What Problem Does This Paper Address?
Spoken dialogue systems (SDS) are becoming increasingly sophisticated, with automatic speech recognition (ASR) achieving high accuracy. However, even with accurate ASR, misunderstandings and non-understandings can occur, leading to dialogue breakdowns. While researchers have focused on optimizing recovery rates (how often the system successfully corrects an error), less attention has been paid to user satisfaction with these recovery strategies. This paper addresses this gap by empirically evaluating how users perceive and prefer different error recovery methods, particularly in the context of misunderstandings. The authors highlight that a misunderstanding can derail a dialogue, and while detection is challenging, user frustration can be a significant factor in overall system usability.
Methods
The authors compare three distinct error recovery strategies:
- Modified MoveOn: This strategy, adapted from Skantze's work, attempts to infer the user's intent without explicitly repeating or confirming each piece of information. Instead, it collects multiple pieces of information and then uses an explicit confirmation for a block of utterances. The core idea is to avoid signaling uncertainty to the user, which can negatively impact their prosody and confidence.
- Reprompt: This is a more basic strategy where, after each dialogue step, the system implicitly confirms the utterance. If a misunderstanding occurs, the user must explicitly signal it (e.g., with "wrong" or "false"), after which the system attempts to correct and then simply reprompts the question.
- Help: Similar to the reprompt strategy, but when a misunderstanding occurs, the system not only attempts to correct and reprompt but also provides additional helpful information related to the current step.
The experiment involved 23 participants, divided into two groups: "computer experts" and "common people." They interacted with the three strategies through short, comparable dialogues written in VoiceXML and processed by the TellMe.Studio platform. To ensure a controlled environment, misunderstandings were faked, and the experiments were conducted under ideal audio conditions with high-quality headsets. User satisfaction was measured using a shortened version of the SASSI questionnaire, focusing on five key factors: Efficiency (EFF), Friendliness (FRI), Reliability (REL), Irritation (IRR), and Mistakes (MIS).
Datasets
Not discussed in the paper. The authors created their own set of dialogues for the experiment.
Key Insights and Ideas
- User satisfaction is a critical metric: While recovery rate is important, how users feel about the recovery process significantly impacts their overall experience and willingness to use the system again.
- "MoveOn" strategy shines: The modified MoveOn strategy, which avoids explicit confirmations and attempts to infer intent, was overwhelmingly preferred by users. This suggests that a system that appears confident and efficient, even when making educated guesses, can lead to a smoother user experience.
- Expert vs. Novice differences: Computer experts generally preferred the efficiency of the MoveOn strategy and were less patient with more verbose strategies. Less experienced users showed more varied reactions, with some struggling more with the system and others being more patient with detailed help.
- The "catch" with MoveOn: While effective, the MoveOn strategy's success relies heavily on its accuracy. If the system's guesses are frequently wrong, it can lead to frustration. The paper notes that in their experiment, the second guess was correct over 90% of the time, which is crucial for its success.
- Gender differences: Women tended to be less bothered and more patient with detailed help prompts than men.
Key Results
- Overall Preference: The Modified MoveOn strategy received the highest satisfaction scores, followed closely by Reprompt, with Help trailing behind.
- Efficiency and Friendliness: Modified MoveOn scored highest in Efficiency and Friendliness.
- Help Strategy Underperformance: The Help strategy performed poorly, potentially due to issues with the barge-in feature of the voice platform, which had to be disabled.
- Expert Users: Experienced users showed a strong preference for the Modified MoveOn strategy, valuing its efficiency.
- Less Experienced Users: This group exhibited greater variability in their responses, with some showing general disapproval of interacting with machines. They were also more irritated by the surprising guesses of the MoveOn system.
Limitations and Future Work
- Faked Misunderstandings: The study used faked misunderstandings, which may not perfectly replicate real-world scenarios where users might exhibit different behaviors.
- Platform Limitations: The reliance on a specific voice platform (TellMe.Studio) and the issues with its barge-in feature may have impacted the results, particularly for the Help strategy.
- Dialogue Design: While efforts were made to keep dialogues comparable, slight differences might have influenced preferences.
- Future Work: The authors propose that future systems should adapt recovery strategies based on user characteristics (background, mood, skill level) to maximize user satisfaction. They suggest that dialogue management systems should actively infer user profiles and adapt accordingly, potentially using prosody features and other cues to inform strategy selection. A policy that considers expected user satisfaction, similar to Bohus and Rudnicky's approach but with a user-centric focus, is recommended.
Summary updated