Proactive Hearing Assistants that Isolate Egocentric Conversations

Guilin Hu, Malek Itani, Tuochao Chen, Shyamnath GollakotaEMNLP 2025

TL;DR

This paper introduces a proactive hearing assistant that automatically identifies and isolates the wearer's conversation partners in real-time, without explicit user prompts. It achieves this by leveraging conversational turn-taking dynamics and the wearer's self-speech as an anchor, enabling a more natural and adaptive hearing experience.

What Problem Does This Paper Address?

The "cocktail party problem" – the difficulty of isolating a desired voice in a noisy, multi-speaker environment – is a significant challenge for individuals with hearing loss, leading to cognitive overload and listening fatigue. Existing hearing aids are often "reactive," requiring manual input to focus on specific sounds. This paper addresses the need for a "proactive" hearing assistant that can automatically infer and enhance the voices involved in the wearer's conversation, adapting to the dynamic nature of real-world social interactions. This is particularly challenging in multi-party conversations where speakers are spatially dispersed and manual enrollment is impractical.

Methods

The core of the proposed system is a dual-model architecture designed for real-time, on-device processing.

  • Anchoring Mechanism: The system uses the wearer's self-speech, extracted via a neural beamformer, as a crucial anchor. This self-speech, combined with turn-taking cues (alternating speech, low overlap, temporal coordination), helps identify conversational partners.
  • Dual-Model Architecture:
    • A fast streaming model operates at a high frequency (12.5 ms chunks) for low-latency extraction of the target conversation.
  • A slower conversational embedding model runs less frequently (every 1 second) to capture longer-range conversational dynamics and discourse structure. This model generates an embedding that conditions the fast streaming model.
  • On-Device Processing: The system is designed to run on embedded devices to minimize communication delays, with the fast model on an Orange Pi 5B and the slower model on an Apple M2 silicon.
  • Spatialization: To train the models for egocentric scenarios, non-egocentric datasets were spatialized using room impulse responses to simulate the acoustic environment of a wearer with binaural microphones.

Datasets

  • Training:
    • Synthetic Datasets: Five synthetic datasets were generated, including variations of LibriTTS (2, 3, 4, 5 speakers, and a "leaving" scenario) combined with RAMC timestamps.
  • Candor: A large dataset of 2-speaker English conversations.
  • RAMC: A dataset of 2-speaker Mandarin conversations.
  • Evaluation:
    • Out-of-Distribution: SpokenWOZ (English 2-speaker), RAMC (Mandarin), and Japanese Duplex Conversation Dataset.
  • Real-world: A newly collected dataset of 6.8 hours of binaural egocentric recordings from 11 participants, featuring 2- and 3-speaker conversations.

Key Insights and Ideas

  • Self-speech as a natural anchor: Using the wearer's own speech as a primary signal to identify who they are talking to is a clever way to avoid explicit user commands and leverage natural conversational behavior.
  • Dual-model for latency and context: The combination of a fast streaming model for immediate response and a slower model for contextual understanding is a pragmatic approach to balance real-time performance with the need to capture complex conversational dynamics.
  • Turn-taking dynamics are key: The system's success hinges on its ability to interpret subtle cues in how people take turns speaking, demonstrating that conversational structure itself can be a powerful signal for speech processing.
  • Generalization across languages: The model's ability to perform well on Mandarin and Japanese conversations despite being trained primarily on English highlights that the learned turn-taking patterns are language-agnostic.

Key Results

  • High Accuracy: The system achieves accuracies of 80-92% in identifying conversational partners across various test sets.
  • Improved Speech Quality: Speech quality of conversation partners is improved by 7.22-11.95 dB (SISDRi).
  • Real-time Performance: The system operates in real-time on embedded devices, with the fast model processing 12.5 ms chunks in an average of 8.9 ms.
  • Adaptation to Dynamics: The model successfully adapts to speakers shifting between target and interfering conversations, as demonstrated by suppressing a speaker after they leave the target conversation.
  • Subjective Improvement: A user study showed a significant improvement in perceived quality, with the overall mean opinion score rising from 1.88 to 4.30.
  • Robustness to Noise: The system shows significant improvements even in noisy conditions, with an SISDRi of 11.84 dB on noisy Libri data.

Limitations and Future Work

  • Active Participant Assumption: The system is designed for scenarios where the wearer is an active participant and uses self-speech. It is not suited for passive listening or eavesdropping.
  • Latency in Simultaneous Speech: The current implementation may struggle with disambiguating overlapping speakers, especially when multiple speakers begin speaking simultaneously. Future work could incorporate lightweight content-aware models.
  • Cultural/Linguistic Nuances: While the model generalizes across languages, further fine-tuning for specific cultural or linguistic turn-taking dynamics might improve robustness.
  • Ethical Considerations: Risks include incorrect speaker detection leading to suppression of relevant voices or amplification of unrelated ones. The authors emphasize the need for transparent design, user-centric controls, and robust real-world evaluation.
  • Wearer Silence: Performance degrades during prolonged periods of wearer silence, highlighting the reliance on self-speech as an anchor. Future work could explore alternative anchoring mechanisms.

Summary updated

BibTeX
@inproceedings{hu2025proactive,
  author = {Guilin Hu and Malek Itani and Tuochao Chen and Shyamnath Gollakota},
  title = {{Proactive Hearing Assistants that Isolate Egocentric Conversations}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2025},
}
Figure