TL;DR
This paper tackles the challenge of separating speech from moving speakers in real-world audio, a problem that often leads to "speaker swaps" where the system inconsistently assigns separated voices to output channels. They propose a novel real-time binaural speech separation system, adapting the Wavesplit network, that uses speaker embeddings and online clustering to reliably track speakers and maintain consistent output assignments, even in long recordings.
What Problem Does This Paper Address?
Imagine you're in a busy cafe, and two people are talking while walking around. Your brain can easily follow who is saying what and where they are. However, current speech separation systems struggle with this. When speakers move, especially in environments with echoes (reverberation), existing methods often get confused. A common issue is the "speaker swap problem": even if the system successfully separates the voices, it might randomly switch which output channel (e.g., left or right ear in headphones) corresponds to which speaker over time. This is particularly problematic for long audio recordings and for applications like augmented hearing, where consistent spatial perception is crucial. Traditional block-wise methods fail to capture enough context, while utterance-level methods can still suffer from inconsistent ordering. This paper aims to solve this by developing a real-time system that can consistently separate and localize moving speakers.
Methods
The core of the proposed solution is an adaptation of the Wavesplit network for binaural speech separation. Here's a breakdown:
- Speaker Profile Module: This module estimates speaker embeddings (numerical representations of a speaker's voice characteristics) for each speaker at every time frame from the mixed binaural audio. Crucially, these embeddings are not inherently ordered.
- Online Clustering: To overcome the unordered embeddings, the system uses online k-means clustering. This algorithm groups similar embeddings together over time and maintains "cluster centroids" which act as stable profiles for each speaker. This effectively tracks speakers even as their embeddings fluctuate.
- Speaker Identification Network (SNet): For training the speaker profile module, they use a pre-trained speaker identification network (SNet). This network provides "oracle" embeddings from clean, individual speaker sources, serving as ground truth during training.
- Permutation Invariant Training (PIT): Frame-level PIT is used during training to align the estimated speaker embeddings with the oracle embeddings, ensuring the system learns to associate the correct embeddings with each speaker.
- Localization and Extraction Modules: The system then uses these speaker profiles to condition a multi-input-multi-output (MIMO) TasNet. This network not only separates the target speaker's audio but also localizes their position (Direction of Arrival - DOA). This is achieved by incorporating interaural phase and level differences (IPD/ILD) into the network's input and using a feature-wise linear modulation (FiLM) mechanism to inject the speaker profile information.
- Causality: A key design choice is that all modules are causal, meaning they process audio in real-time without needing future information, making it suitable for low-latency applications.
Datasets
- Training and Validation: The system was trained and validated using simulated binaural room impulse responses (BRIRs) with varying reverberation times (RT60 from 0 to 0.7 seconds) and 26.7 hours of training data and 6.7 hours of validation data.
- Testing: For testing, they used:
- Simulated room data (4 hours).
- Real room data from three rooms with different RT60 values (1 hour total).
- Speech data was sourced from the Librispeech dataset.
- Mixtures were created with two moving speakers, random relative Signal-to-Noise Ratios (SNRs) between 0 and 5 dB, and random movement velocities and directions.
Key Insights and Ideas
- Speaker Embeddings as Dynamic Profiles: The paper highlights the power of using dynamic speaker embeddings that are updated over time to represent moving speakers, rather than relying on static speaker models.
- Online Clustering for Robust Tracking: Online k-means clustering proves to be an effective and computationally efficient way to maintain consistent speaker tracking in real-time, mitigating the speaker swap problem.
- Conditioning Separation with Speaker Profiles: Conditioning a separation network (TasNet) with speaker profiles derived from embeddings significantly improves robustness to speaker swaps and enhances separation quality.
- Joint Localization and Separation: The integration of a localization module that predicts DOA, guided by speaker profiles, alongside the separation module, allows for a more holistic understanding of the acoustic scene.
- Generalization to Real Rooms: The model, trained on simulated data, demonstrates good generalization capabilities to real-world room acoustics, even with significant reverberation.
Key Results
- Reduced Speaker Swaps: The proposed SPK-MIMO TasNet significantly reduces speaker swaps compared to uPIT-MIMO TasNet, especially in long recordings (e.g., 0.6 swaps vs. 3.4 swaps on 24-second recordings).
- Competitive Separation Performance: The system achieves comparable performance to uPIT-MIMO TasNet with ground truth tracking in terms of Signal-to-Noise Ratio (SNR) and Direction of Arrival (DOA) error. For instance, on 2.4-second recordings, it achieved an 8.3 dB SNR and 8.2° DOA error, very close to the oracle-conditioned version.
- Impact of Post-Enhancement: Applying a post-binaural speech enhancement module further improves SNR and reduces DOA error for both uPIT-MIMO TasNet and the proposed method.
- Reverberation Impact: While stronger reverberation degrades separation quality (SNR) and spatial cue preservation (DOA error), it has a minimal impact on the number of speaker swaps.
Limitations and Future Work
- Fixed Number of Speakers: The current system assumes a fixed number of speakers (N=2). Future work could explore handling an unknown or changing number of speakers.
- Training Data: While generalization to real rooms is shown, further work could involve training on more diverse real-world data.
- Model Efficiency: The authors suggest developing more efficient and compact models for deployment on hearable devices with limited power and storage.
- Handling Unknown Speakers: The system is speaker-independent but relies on learned speaker profiles. Adapting it to handle entirely novel speakers not seen during training could be an area for future research.
Summary updated