Introduction
The "always listening" paradigm in modern voice assistants presents a significant challenge in multi-user environments. In Human-Human-Computer (H-H-C) scenarios—such as two people using a voice-controlled interface together—systems must accurately distinguish between speech directed at the computer and speech directed at other humans. This task, known as addressee detection, is complicated by the fact that natural language commands often overlap lexically with human-to-human conversation, and traditional triggers like keywords or gaze are often unreliable in shared social spaces.
Methodology
This research explores the fusion of lexical and acoustic-prosodic features to improve addressee detection accuracy.
Lexical Features
Lexical models utilize word sequences (N-grams) derived from both Automatic Speech Recognition (ASR) and human-corrected reference transcripts. These models also incorporate context-aware features, such as the "maximum cosine similarity" between the utterance and on-screen content, alongside ASR confidence scores to assess the reliability of the linguistic input.
Acoustic-Prosodic Features
To overcome the limitations of lexical analysis—particularly in noisy environments or with unconstrained, free-form speech—the authors introduced word-independent and speaker-independent prosodic features:
- Segment-level Statistics: Metrics including speaking rate, energy peaks, and pause durations.
- Energy Contour Models: Recognizing that humans often adopt a rhythmic, "sing-songy" tone when addressing machines, the researchers used Gaussian Mixture Models (GMMs) to capture the "shape" of speech volume over 200ms sliding windows. This allows the system to recognize the acoustic signature of computer-directed speech without needing to decode the actual words.
These models were integrated using Linear Logistic Regression (LLR) to produce a unified detection score.
Dataset
The study utilized a specialized H-H-C corpus consisting of 6.3 hours of recordings from 17 sessions. The dataset includes 1,802 speech segments from 13 unique speakers interacting with a dialog system via a Kinect microphone while seated in front of a large television. Utterances were manually categorized as "Computer-directed" (commands or free-form), "Human-directed," or "Mixed."
Results and Analysis
The performance was evaluated using the Equal Error Rate (EER), the point where false alarms and misses are balanced.
As shown in the results table, the prosodic-only model (System 5, 12.63% EER) significantly outperformed the lexical-only ASR model (System 2, 23.11% EER). The fusion of both feature sets (System 6) further reduced the EER to 11.08%. Even in a scenario with perfect speech recognition (System 7 vs. System 8), adding prosodic features yielded a 35% relative error reduction, dropping the EER from 10.16% to 6.72%.
A key finding is the effectiveness of prosody in filtering out human-to-human speech. While lexical models frequently misclassify human-directed talk as system commands, the prosodic models are much more adept at identifying the unique rhythmic signature of "computer-directed" speech. The most dramatic improvement was observed in the "Human-directed" category: the best lexical model had a 71% error rate on human-to-human speech, which the prosodic model slashed to 33%.
Conclusion
This work demonstrates that the subconscious shift in prosody when addressing machines is a robust signal for addressee detection. By combining linguistic cues with energy contour modeling, systems can significantly reduce false activations in complex, multi-user environments, even when faced with imperfect speech recognition. Future work will focus on integrating pitch features and evaluating these models in real-time, low-latency applications.
Summary updated