Introduction
Modern communication technology is predominantly user-centric, designed for individuals interacting via personal devices. The Together Anywhere, Together Anytime (TA2) project seeks to shift this paradigm toward "group-centric" communication, enabling spatially separated families to interact naturally in a shared "living room" environment. Achieving this requires a "virtual director"—an automated system capable of managing audio-visual streams based on real-time social context.
The primary challenge lies in the acoustic complexity of residential spaces, which are often reverberant and noisy. To function effectively, the system must identify who is speaking, their location, and the content of their speech with minimal latency to maintain the feeling of "being there."
System Architecture
The researchers developed a modular architecture built upon two core frameworks: Tracter, a lightweight data-flow framework, and Juicer, a Weighted Finite State Transducer (WFST) based ASR decoder.
Tracter utilizes a "pull" mechanism, where the end of the processing chain requests data from the source. This design ensures that the most computationally intensive components, such as the ASR decoder, dictate the processing speed, preventing system lag and ensuring real-time performance.
Technical Implementation
The system captures audio using specialized hardware, such as diamond-shaped 4-microphone or circular 8-microphone arrays. To maintain low latency, audio is streamed via network sockets in 10ms packets, keeping transmission latency below 20ms.
Direction of Arrival (DOA) and Voice Activity
Speaker localization is achieved through the Sector-based Activity Measure (SAM) algorithm. By employing a particle filter to cluster sound sources, the system can track multiple concurrent speakers even in reverberant conditions. Audio signals are processed into 39-dimensional feature vectors (Mel-cepstrum coefficients with velocity and acceleration derivatives).
A critical innovation is the Low-Delay Association module. Rather than waiting for full sentence completion, the system associates directional data with voice activity within a tight 150–300ms window. This allows the virtual director to react almost instantaneously to changes in the conversation.
Evaluation and Results
The system's performance was validated using standard benchmarks (NIST RT07/RT09) and real-world datasets from Fraunhofer IIS.
Localization Precision
In controlled tests with fixed sound sources, the system demonstrated high spatial accuracy. For speakers positioned at ±45°, the Direction of Arrival estimation achieved a standard deviation of only 2.3°–2.4°.
Even as distance and reverberation increased, the error margin remained below 5°, providing reliable data for automated camera switching.
Latency and ASR Performance
Transmission latency was successfully optimized to 12–20 ms. While far-field ASR remains challenging—yielding Word Error Rates (WER) of 41% for distant microphones—the system compensates by focusing on keyword and proper name spotting. This metadata provides sufficient context for the virtual director to make informed decisions without requiring perfect transcription.
Conclusion
By prioritizing low-latency metadata over perfect verbatim transcription, the Idiap Research Institute has created a robust framework for group-to-group communication. Future work aims to incorporate multimodal fusion, combining audio data with face tracking to improve speaker identification in noisy or crowded environments.
Summary updated