Abstract: The dialogue strategies used by a spoken dialogue system strongly influence performance and user satisfaction. An ideal system would not use a single fixed strategy, but would adapt to the circumstances at hand. To do so, a system must be able to identify dialogue properties that suggest adaptation. This paper focuses on identifying situations where the speech recognizer is performing poorly. We adopt a machine learning approach to learn rules from a dialogue corpus for identifying these situations. Our results show a significant improvement over the baseline and illustrate that both lower-level acoustic features and higher-level dialogue features can affect the performance of the learning algorithm.
TL;DR
Researchers have developed a machine learning approach to automatically detect when a spoken dialogue system is failing at the "dialogue level" rather than just the "utterance level." By training a classifier on features ranging from low-level acoustic confidence to high-level dialogue behaviors, the system achieved 77.4% accuracy in identifying "bad" dialogues, providing a vital trigger for systems to dynamically adapt their strategies.
The Challenge of Dialogue-Level Detection
Spoken dialogue systems, such as automated phone menus or early voice assistants, often struggle with speech recognition errors. Traditionally, these systems employ a fixed strategy—for example, always asking open-ended questions or requiring explicit confirmation for every step. However, an ideal system should be adaptive: if it realizes it is failing to understand the user, it should switch to a more robust strategy, such as a touch-tone menu or more restrictive prompts.
The primary challenge is detection. While systems often generate a "confidence score" for a single sentence, a single misrecognition does not necessarily mean the entire conversation is failing. Conversely, a series of seemingly acceptable scores might mask a frustrated user who is stuck in a loop or unable to complete their task.
This research addresses the need to identify "poor recognition performance" across an entire dialogue using automated logs, allowing the system to decide when it is time to change tactics.
Methodology: Features and Rule Induction
The authors treated the problem as a supervised binary classification task: labeling a dialogue as either "good" or "bad." A dialogue was labeled "bad" if more than 11% of the user's utterances were misrecognized, based on human-annotated ground truth.
The RIPPER Algorithm
The researchers utilized RIPPER, a machine learning program that induces human-readable "if-then" rules. Unlike "black box" models, RIPPER produces rules that are easy for developers to interpret and implement, such as: “If the user cancels more than 6% of the time, the dialogue is bad.”
Feature Engineering
They extracted 23 primitive features from system logs, categorized into five distinct types:
- Acoustic: Mean ASR confidence scores and predicted misrecognition rates.
- Dialogue Efficiency: Total elapsed time, number of system turns, and user turns.
- Dialogue Quality: Frequency of "rejections" (system didn't understand), timeouts, help requests, cancellations, and "barge-ins" (user interrupting the system).
- Experimental Parameters: The specific system used, the task, and the user's expertise level.
- Lexical: The actual recognized text (e.g., did the user say "help" or "cancel"?).
Experimental Setup
The study utilized a corpus of 544 dialogues (representing over 40 hours of speech) collected from three distinct AT&T Labs systems: ANNIE (voice dialing), ELVIS (email access), and TOOT (train schedules).
Results: The Impact of Feature Integration
The researchers compared various feature combinations against a baseline accuracy of 52% (the accuracy achieved by simply guessing "good" every time). The results demonstrated that while individual feature sets provided improvements, the combination of all features yielded the most robust detection.
Key findings from the results include:
- High-Level Features are Predictive: Dialogue quality features (like help requests) were nearly as predictive as low-level acoustic confidence scores.
- Redundancy as a Strength: Different feature sets (acoustic vs. lexical) often yielded similar accuracy, suggesting that systems lacking sophisticated acoustic measures can still detect failure by monitoring user behavior.
- The Power of Context: The "All Features" model performed best (77.4% accuracy) because it could combine acoustic data with dialogue flow. For instance, a low confidence score combined with a "cancel" command is a much stronger signal of failure than either feature alone.
Conclusion and Future Directions
The jump from a 52% baseline to 77.4% accuracy represents a significant improvement, proving that system logs contain enough "signal" to identify failing conversations automatically.
Future work aims to move beyond post-hoc analysis to real-time, mid-dialogue classification. Additionally, the researchers suggest incorporating "prosodic" features (like tone of voice or stress levels) and direct measures of user satisfaction to further refine the system's ability to adapt to user needs.
Summary updated