TL;DR
This paper explores how a spoken dialog system can adapt to a user's speaking style (shouting or hyperarticulation) to improve speech recognition and task success. They developed detectors for these styles and implemented three strategies (explicit, implicit, backoff) to encourage users to revert to "unmarked" speech, finding that explicit instructions were most effective.
What Problem Does This Paper Address?
Human-to-human conversations often involve subtle adaptations in speaking style, a phenomenon known as entrainment. This paper investigates whether spoken dialog systems can also engage in this kind of prosodic entrainment. Specifically, they focus on two challenging speaking styles: shouting and hyperarticulation. These styles significantly degrade Automatic Speech Recognition (ASR) performance, and frustrating dialogs can even lead users to adopt these styles more frequently. The core problem is how to make dialog systems more robust to these non-standard speaking styles and, more importantly, how to guide users back to a more recognizable speech pattern to improve overall dialog success.
Methods
The authors developed real-time detectors for shouting and hyperarticulation using features extracted by the openSMILE toolkit. These features included acoustic characteristics like fundamental frequency range and intensity, as well as dialog-level features such as system confidence and explicit confirmation repetition. They trained Support Vector Machines (SVMs) on an expert-labeled dataset of user turns from the "Let's Go" spoken dialog system.
To address the detected styles, three distinct dialog strategies were implemented:
- Explicit: The system directly instructs the user to change their speaking style (e.g., "I'm sorry. I cannot understand loud speech very well, please speak softer.").
- Implicit: The system attempts to entrain the user by demonstrating the desired behavior. For shouting, this meant lowering the Text-to-Speech (TTS) volume; for hyperarticulation, it meant increasing the TTS speaking rate.
- Backoff: The system changes the dialog topic or slot objective, aiming to provide a context switch that might naturally lead the user to alter their speaking style.
If a strategy fails to elicit a change, the system randomly selects another unused strategy.
Datasets
The study utilized a dataset of 2259 call turns from the "Let's Go" spoken dialog system, collected in 2009 and 2010, which was expert-labeled for shouting and hyperarticulation. For the strategy evaluation, 1528 dialogs were collected over 28 days in February and March 2012.
Key Insights and Ideas
- Prosody Matters: The paper reinforces that prosodic variations like shouting and hyperarticulation significantly impact ASR performance and dialog success, not just at the turn level but also across the entire dialog.
- Explicit is Effective: Directly telling users to change their speaking style proved to be the most effective strategy for both shouting and hyperarticulation, suggesting that users are receptive to clear instructions.
- Shouting is More Malleable: Shouting was found to be more amenable to modification by the system's strategies compared to hyperarticulation. This might be due to shouting being a more conscious or controllable style.
- Hyperarticulation is Tricky: The authors suggest hyperarticulation is a "loose class definition" and potentially a subconscious behavior, making it harder for both detectors and users to control.
- Implicit Entrainment Works, But...: While the implicit strategy showed some success, it was less effective than the explicit approach, indicating that while users can entrain to system behavior, direct guidance is more potent.
- Backoff as a Context Switch: The backoff strategy, by changing the dialog context, also showed promise in altering user speaking styles.
Key Results
- Detector Performance: The shouting detector achieved 80% accuracy, and the hyperarticulation detector achieved 70% accuracy on a balanced test set.
- Turn-Level Impact: Shouting increased Word Error Rate (WER) by +53.6% and decreased ASR confidence by -7.9%. Hyperarticulation increased WER by +11.9% and decreased ASR confidence by -13.7%.
- Dialog-Level Impact: Shouting and hyperarticulation significantly increased dialog length and the number of turns. Shouting statistically significantly lowered estimated success rates, while hyperarticulation's presence also decreased success, especially with higher percentages of turns in that style.
- Strategy Efficacy: The explicit strategy showed the most significant improvement in reverting to "unmarked" speech: +74.3% for shouting and +17.9% for hyperarticulation over the baseline. Implicit and backoff strategies also showed improvements, with shouting being more responsive to implicit strategies.
- Overall System Impact: Strategies reduced instances of shouting and hyperarticulation from 25.8% to 23.0%. While strategies slightly increased dialog duration, they reduced the number of turns per dialog.
Limitations and Future Work
- Detector Accuracy: The detectors, while achieving reasonable accuracy, are not perfect. Residual errors and the assumption that detectors might make similar mistakes across strategies could influence results. The lower accuracy for hyperarticulation detection is a specific concern.
- Hyperarticulation Definition: The broad and potentially subconscious nature of hyperarticulation makes it a challenging style to define, detect, and modify. The authors note low inter-annotator agreement for this style.
- Task Success vs. Style Change: While strategies effectively changed speaking styles, their impact on overall task success was relatively small, except for the explicit/backoff combination for shouting.
- Future Work: The authors suggest that the inclusion of these strategies, even with their modest impact on task success, can improve system performance and perceived naturalness. Further research could explore other speaking styles and refine the existing strategies.
Summary updated