TL;DR
This paper tackles the challenge of improving spoken dialogue systems by automatically identifying and understanding user corrections. By analyzing acoustic features of user utterances following system errors, the authors demonstrate that distinct speaking styles emerge for different types of corrections, enabling the development of decision trees that can distinguish corrections from original inputs with high accuracy.
What Problem Does This Paper Address?
Miscommunication is a significant hurdle for the widespread adoption of speech recognition systems. When these systems make errors (misrecognitions or rejections), users often attempt to correct them. However, current systems struggle to reliably detect when a correction is happening and to accurately process the content of these correction utterances. This difficulty in handling corrections directly impacts user satisfaction and the overall usability of voice-based interfaces. The paper aims to bridge this gap by characterizing the acoustic differences between original user inputs and their subsequent corrections, and by developing a method to automatically identify these corrections.
Methods
The researchers analyzed over 300 pairs of original and repeat user utterances from a telephone-based voice-in/voice-out conversational system. They extracted a range of acoustic-prosodic features, including:
- Durational: Total utterance duration, speaking rate.
- Pause: Number of pauses, average pause duration, total pause duration, and silence as a percentage of total utterance duration.
- Pitch: Pitch maximum, minimum, range, final pitch contour, and slopes of pitch rise and fall.
- Amplitude: Mean amplitude, amplitude of the last voiced region, amplitude of the loudest region, standard deviation, and differences from mean/last/maximum.
These features were then used to train decision trees (specifically, Quinlan's C4.5 algorithm) to classify utterances. The goal was to distinguish between:
- Original inputs vs. repeat corrections.
- Corrections of misrecognition errors (CME) vs. corrections of rejection errors (CRE).
- CME's vs. original inputs.
Datasets
The study utilized approximately 60 hours of user interactions from a field trial of the Sun Microsystems SpeechActs system. This dataset comprised user interactions with various desktop applications like email, calendar, and weather reports. The data was recorded at 8kHz sampling in 8-bit mu-law encoding. User utterances were transcribed, and recognition results were logged. Each utterance was then labeled with one of four accuracy codes: OK (correct recognition, correct action), Error Minor (near-exact recognition, correct action), Error (incorrect recognition, incorrect action), and Rejection (no recognition result, no action). The paper specifically focused on pairs of original and repeat utterances following an "Error" or "Rejection" code.
Key Insights and Ideas
- Corrections are acoustically distinct: Users adopt a different speaking style when correcting errors compared to their initial inputs. This shift is not uniform across all correction types.
- CME vs. CRE divergence: Corrections of misrecognition errors (CME) show more pronounced acoustic changes than corrections of rejection errors (CRE). This suggests that users employ different strategies depending on whether the system misunderstood them or failed to produce any output.
- Pitch variability as a key differentiator: CME's exhibit significantly heightened pitch variability, particularly in the slope of pitch rise, which is a strong indicator of contrastive intent. CRE's, on the other hand, show a slight decrease in pitch minimum.
- Duration and pause are universal indicators: Both CME's and CRE's tend to be longer and contain more pauses than original inputs, aligning with a general shift towards clearer, more deliberate speech.
- Decision trees offer an interpretable solution: The use of decision trees allows for the identification of the most relevant acoustic features, providing an interpretable model for distinguishing correction types without requiring perfect textual transcriptions.
Key Results
- Duration and Pause: Both total utterance duration and total pause length significantly increase from original inputs to repeat corrections. This increase is more pronounced for CME's than for CRE's.
- Pitch:
- CRE's showed a small but significant decrease in pitch minimum for male speakers.
- CME's displayed a large and significant increase in pitch variability (measured by steepest rise slope) compared to original inputs.
- Decision Tree Performance:
- Classifying original vs. repeat corrections achieved an error rate of 33%.
- Distinguishing CME's from CRE's had an error rate of 30.7%.
- The most successful classifiers separated original inputs from CME's, achieving an accuracy rate of 75% (25% error). These trees relied heavily on pitch slope and durational features.
Limitations and Future Work
Limitations: * The study focused on a specific telephone-based system and may not generalize perfectly to all voice-in/voice-out systems or different acoustic environments. * While the decision trees are interpretable, the "best" results for distinguishing CME's from CRE's still had a 30.7% error rate, indicating room for improvement.
Future Work: * The authors plan to use these findings to adapt standard recognition procedures to improve recognition accuracy on spoken corrections. * They aim to develop more accurate characterizations of original vs. correction inputs to better handle error correction interactions. * The ultimate goal is to ease recovery from human-computer miscommunication and lower the barrier to widespread acceptance of spoken language systems.
Summary updated