TL;DR
This paper introduces a novel approach to addressee detection in dialog systems by analyzing speaking style, specifically rhythm and vocal effort, using only acoustic features. This method significantly outperforms traditional word-based approaches and shows promising generalization to new domains, paving the way for more robust and natural human-computer interaction.
What Problem Does This Paper Address?
As dialog systems become more sophisticated and are deployed in open environments where multiple people might be present, it's crucial for them to distinguish between speech directed at the system and speech directed at other humans. This "addressee detection" (AD) problem is essential for seamless human-computer interaction, especially in scenarios involving multiple users and the system. Previous AD research often relied on speech recognition output, which can be error-prone, or required extensive speaker and session normalization, limiting its applicability in real-world, unconstrained settings. This paper aims to address this by developing AD methods that are independent of speech recognition, speaker characteristics, and dialog context, focusing instead on inherent speaking style.
Methods
The core innovation of this paper lies in its use of acoustic-prosodic features derived from speaking style, specifically rhythm and vocal effort.
- Rhythmicity: This is captured using "energy contour DCT" (encon) features. These features model the rhythm of speech by analyzing the temporal patterns of energy in the speech signal over short windows. Importantly, these features do not require speech recognition or normalization.
- Vocal Effort: To capture vocal effort, the researchers developed "voice quality and spectral tilt" (tilt) features and "delta energy at voicing onsets/offsets" (devo) features. These features analyze spectral characteristics and energy changes around voiced segments to infer how much effort the speaker is exerting. This approach is designed to work without direct energy normalization, a key constraint.
These acoustic features are then fed into either Gaussian Mixture Models (GMMs) or an adaptive boosting algorithm for classification. The paper also explores combining these prosodic features with traditional lexical features (derived from automatic speech recognition) and evaluates their performance in simulated online processing scenarios and out-of-domain training settings.
Datasets
The primary dataset used in this research is a newly collected corpus of human-human-computer (HHC) interactions. This corpus consists of approximately 6.3 hours of recordings from 38 sessions involving two speakers interacting with a "Conversational Browser" (CB) dialog system. The data was hand-transcribed and annotated for addressee (human-directed, computer-directed, or mixed).
For out-of-domain training experiments, the researchers utilized existing corpora:
- Human-Computer (H-C) corpora: ATIS and Communicator.
- Human-Human (H-H) corpora: Fisher and the ICSI Meeting corpus.
Key Insights and Ideas
- Speaking Style as a Powerful AD Cue: The research strongly demonstrates that subtle changes in speaking style, like rhythm and vocal effort, are highly indicative of the addressee. This is a significant departure from relying solely on the content of speech.
- Independence is Key: Features that do not require speech recognition, speaker normalization, or dialog context are more robust and practical for real-world applications. This paper successfully develops such features.
- Complementarity of Features: Rhythm and vocal effort features, while distinct, are complementary. Combining them leads to a significant improvement in addressee detection accuracy.
- Early Detection is Possible: Acoustic-prosodic features enable addressee detection very early in an utterance (within the first 2 seconds of speech), which can be leveraged to reduce system latency and improve user experience.
- Generalization Potential: Acoustic-prosodic features show promising generalization to out-of-domain corpora, even those containing only H-C or H-H interactions. This suggests that these style-based cues are not domain-specific and can be learned from broader datasets.
Key Results
- The "all-prosody" system, combining energy contour, spectral tilt, and delta energy features, achieved an Equal Error Rate (EER) of 12.5%. This significantly outperforms the non-cheating word N-gram system (27.0% EER).
- When combined with lexical features, the "all-prosody" system achieved an EER of 10.9%, demonstrating the benefit of integrating content and style information.
- Simulated online processing showed that most systems achieve good performance within the first 2 seconds of speech.
- Out-of-domain training using ATIS (H-C) and ICSI (H-H) data, when combined, yielded an EER of 22.8%, only slightly higher than the in-domain trained system, highlighting good generalization.
Limitations and Future Work
- Limitations: While the paper demonstrates strong results, it acknowledges that the "cheating" N-gram (using reference transcripts) still achieves the best performance, indicating that lexical information is valuable when available and accurate. The out-of-domain generalization, while promising, still shows a performance drop compared to in-domain training.
- Future Work: The authors suggest that further exploration of acoustic-prosodic features and their combination with lexical models could lead to even better performance. The promising generalization of these features also opens avenues for developing domain-independent AD systems, which would be highly beneficial for broad deployment of dialog systems.
Summary updated