TL;DR
This paper analyzes how users behave when spoken dialog systems make errors, using real-world data from the DARPA Communicator project. They found that how a user discovers an error significantly impacts recovery time and success, and that successful recovery often involves rephrasing or starting over, rather than trying to fix a chain of errors.
What Problem Does This Paper Address?
Spoken dialog systems, while increasingly sophisticated, are prone to errors, primarily from Automatic Speech Recognition (ASR). These errors can disrupt the user's experience and lead to task failure. Understanding how users react to these errors is crucial for designing more robust and user-friendly dialog systems. This paper specifically investigates user behavior under various error conditions, aiming to model these reactions and identify strategies that lead to successful task completion. The challenge lies in moving beyond simply detecting errors to understanding the nuanced dynamics of user recovery and the factors influencing it.
Methods
The researchers analyzed a corpus of transcribed travel arrangement dialogs from the DARPA Communicator project. They developed a detailed manual tagging scheme to categorize user behavior, system behavior, and task status, focusing on error perception and recovery. A key aspect of their methodology was to analyze user behavior conditioned on system behavior, using a smoothed conditional probability model. This model incorporated the ASR word error rate (WER) to account for the severity of errors, mapping WER to a range between 0 and 1 for smoothing. They also examined user initiative in both error and non-error environments.
Datasets
The study utilized a portion of the June 2000 DARPA Communicator dialogs. This dataset comprised approximately 141 dialogs involving 85 experimental subjects interacting with 9 different "travel agent" systems. The data was manually transcribed and annotated with a custom tagging scheme. This dataset is a valuable resource for studying human-computer interaction in spoken dialogs, particularly under error conditions.
Key Insights and Ideas
- Error Discovery Matters: The way a user becomes aware of an error has a significant impact on recovery. Implicit confirmations and system rejections are common, but implicit confirmations lead to longer recovery times and lower success rates compared to explicit system feedback.
- Successful Recovery Strategies: Users who successfully recover from errors tend to rephrase or repeat their requests more often than those who don't. They are also more likely to "start over" or "scratch" the current attempt rather than trying to repair a sequence of errors.
- User Initiative Varies: User initiative (acting beyond simply responding to prompts) is significantly higher in error segments compared to non-error segments, suggesting users become more proactive when things go wrong.
- Individual Differences: There are notable individual differences in error recovery success. Some users consistently recover well, even with a higher number of errors, while others struggle, experiencing longer error episodes or failing to get back on track.
Key Results
- Approximately 80% of error segments in the dataset were between 1-9 turns long.
- Implicit confirmation as an error discovery method resulted in a longer average error segment length (9.6 turns) and a lower back-on-track (BOT) rate (68%) compared to system rejection (6 turns, 83% BOT) or system repeat (5.8 turns, 90% BOT).
- In successful error recoveries, users were significantly more likely to rephrase and less likely to contradict. They also favored "start over" and "scratch" strategies.
- User initiative behaviors like "ask" and "contradict" were more frequent in error segments (0.0319 and 0.0707 respectively) than in non-error segments (0.0060 and 0.0121 respectively).
Limitations and Future Work
The authors acknowledge that the experimental setup involved paid participants with no real stake in task completion, which might influence their behavior (e.g., some may have been inattentive). They also note that some participants had more difficulty producing usable ASR output. Future work is suggested to incorporate user behavior priors into system models, as some users appear inherently better at navigating errors. The ongoing research aims to explore how user models interact with system models for optimization.
Summary updated