Learning When to Listen: Detecting System-Addressed Speech in Human-Human-Computer Dialog

Elizabeth Shriberg, Andreas Stolcke, Dilek Hakkani-Tür, Larry P. HeckInterspeech 2012 · OpenReview

Introduction

The "always listening" paradigm in modern voice assistants presents a significant challenge in multi-user environments. In Human-Human-Computer (H-H-C) scenarios—such as two people using a voice-controlled interface together—systems must accurately distinguish between speech directed at the computer and speech directed at other humans. This task, known as addressee detection, is complicated by the fact that natural language commands often overlap lexically with human-to-human conversation, and traditional triggers like keywords or gaze are often unreliable in shared social spaces.

Methodology

This research explores the fusion of lexical and acoustic-prosodic features to improve addressee detection accuracy.

Lexical Features

Lexical models utilize word sequences (N-grams) derived from both Automatic Speech Recognition (ASR) and human-corrected reference transcripts. These models also incorporate context-aware features, such as the "maximum cosine similarity" between the utterance and on-screen content, alongside ASR confidence scores to assess the reliability of the linguistic input.

Acoustic-Prosodic Features

To overcome the limitations of lexical analysis—particularly in noisy environments or with unconstrained, free-form speech—the authors introduced word-independent and speaker-independent prosodic features:

  • Segment-level Statistics: Metrics including speaking rate, energy peaks, and pause durations.
  • Energy Contour Models: Recognizing that humans often adopt a rhythmic, "sing-songy" tone when addressing machines, the researchers used Gaussian Mixture Models (GMMs) to capture the "shape" of speech volume over 200ms sliding windows. This allows the system to recognize the acoustic signature of computer-directed speech without needing to decode the actual words.

These models were integrated using Linear Logistic Regression (LLR) to produce a unified detection score.

Dataset

The study utilized a specialized H-H-C corpus consisting of 6.3 hours of recordings from 17 sessions. The dataset includes 1,802 speech segments from 13 unique speakers interacting with a dialog system via a Kinect microphone while seated in front of a large television. Utterances were manually categorized as "Computer-directed" (commands or free-form), "Human-directed," or "Mixed."

Results and Analysis

The performance was evaluated using the Equal Error Rate (EER), the point where false alarms and misses are balanced.

Performance comparison of lexical, prosodic, and fused models across different system configurations.

As shown in the results table, the prosodic-only model (System 5, 12.63% EER) significantly outperformed the lexical-only ASR model (System 2, 23.11% EER). The fusion of both feature sets (System 6) further reduced the EER to 11.08%. Even in a scenario with perfect speech recognition (System 7 vs. System 8), adding prosodic features yielded a 35% relative error reduction, dropping the EER from 10.16% to 6.72%.

Error rate breakdown by addressee category (Computer command, Computer non-command, Human, Mixed) across different system configurations.

A key finding is the effectiveness of prosody in filtering out human-to-human speech. While lexical models frequently misclassify human-directed talk as system commands, the prosodic models are much more adept at identifying the unique rhythmic signature of "computer-directed" speech. The most dramatic improvement was observed in the "Human-directed" category: the best lexical model had a 71% error rate on human-to-human speech, which the prosodic model slashed to 33%.

Conclusion

This work demonstrates that the subconscious shift in prosody when addressing machines is a robust signal for addressee detection. By combining linguistic cues with energy contour modeling, systems can significantly reduce false activations in complex, multi-user environments, even when faced with imperfect speech recognition. Future work will focus on integrating pitch features and evaluating these models in real-time, low-latency applications.

Summary updated

BibTeX
@inproceedings{shriberg2012learning,
  author = {Elizabeth Shriberg and Andreas Stolcke and Dilek Hakkani-Tür and Larry P. Heck},
  title = {{Learning When to Listen: Detecting System-Addressed Speech in Human-Human-Computer Dialog}},
  booktitle = {Interspeech},
  year = {2012},
  url = {https://openreview.net/forum?id=2AgKRxbssCm},
}
Figure