Hands free audio analysis from home entertainment

Danil Korchagin, Philip N. Garner, Petr MotlícekInterspeech 2010 · OpenReview

Introduction

Modern communication technology is predominantly user-centric, designed for individuals interacting via personal devices. The Together Anywhere, Together Anytime (TA2) project seeks to shift this paradigm toward "group-centric" communication, enabling spatially separated families to interact naturally in a shared "living room" environment. Achieving this requires a "virtual director"—an automated system capable of managing audio-visual streams based on real-time social context.

The primary challenge lies in the acoustic complexity of residential spaces, which are often reverberant and noisy. To function effectively, the system must identify who is speaking, their location, and the content of their speech with minimal latency to maintain the feeling of "being there."

System Architecture

The researchers developed a modular architecture built upon two core frameworks: Tracter, a lightweight data-flow framework, and Juicer, a Weighted Finite State Transducer (WFST) based ASR decoder.

High-level system architecture showing the MasterController and its interfaces with audio and network components.

Tracter utilizes a "pull" mechanism, where the end of the processing chain requests data from the source. This design ensures that the most computationally intensive components, such as the ASR decoder, dictate the processing speed, preventing system lag and ensuring real-time performance.

Technical Implementation

The system captures audio using specialized hardware, such as diamond-shaped 4-microphone or circular 8-microphone arrays. To maintain low latency, audio is streamed via network sockets in 10ms packets, keeping transmission latency below 20ms.

Direction of Arrival (DOA) and Voice Activity

Speaker localization is achieved through the Sector-based Activity Measure (SAM) algorithm. By employing a particle filter to cluster sound sources, the system can track multiple concurrent speakers even in reverberant conditions. Audio signals are processed into 39-dimensional feature vectors (Mel-cepstrum coefficients with velocity and acceleration derivatives).

A critical innovation is the Low-Delay Association module. Rather than waiting for full sentence completion, the system associates directional data with voice activity within a tight 150–300ms window. This allows the virtual director to react almost instantaneously to changes in the conversation.

Evaluation and Results

The system's performance was validated using standard benchmarks (NIST RT07/RT09) and real-world datasets from Fraunhofer IIS.

Localization Precision

In controlled tests with fixed sound sources, the system demonstrated high spatial accuracy. For speakers positioned at ±45°, the Direction of Arrival estimation achieved a standard deviation of only 2.3°–2.4°.

Graph showing the precision of Direction of Arrival (DOA) estimation as sound sources switch between ±45°.

Even as distance and reverberation increased, the error margin remained below 5°, providing reliable data for automated camera switching.

Latency and ASR Performance

Transmission latency was successfully optimized to 12–20 ms. While far-field ASR remains challenging—yielding Word Error Rates (WER) of 41% for distant microphones—the system compensates by focusing on keyword and proper name spotting. This metadata provides sufficient context for the virtual director to make informed decisions without requiring perfect transcription.

Conclusion

By prioritizing low-latency metadata over perfect verbatim transcription, the Idiap Research Institute has created a robust framework for group-to-group communication. Future work aims to incorporate multimodal fusion, combining audio data with face tracking to improve speaker identification in noisy or crowded environments.

Summary updated

BibTeX
@inproceedings{korchagin2010hands,
  author = {Danil Korchagin and Philip N. Garner and Petr Motlícek},
  title = {{Hands free audio analysis from home entertainment}},
  booktitle = {Interspeech},
  year = {2010},
  url = {https://openreview.net/forum?id=xherXE16A9N},
}
Figure