Real-Time Binaural Speech Separation with Preserved Spatial Cues

Cong Han, Cong Han 0001, Yi Luo, Nima Mesgarani

TL;DR

Researchers at Columbia University have developed MIMO TasNet, a real-time speech separation system that isolates individual voices from a mixture while perfectly preserving their 3D spatial location. By moving beyond traditional "mono" separation to a multi-input-multi-output (MIMO) architecture, the system maintains the subtle timing and volume differences between ears, paving the way for "intelligent" hearing aids and immersive augmented reality.

What Problem Does This Paper Address?

Most deep learning speech separation models are "Multi-Input Single-Output" (MISO). They take audio from multiple microphones but produce a single, mono-channel output for each speaker. While the speech is clear, the spatial cues—the "where" of the sound—are lost.

For humans, spatial hearing relies on two primary cues:

  1. Interaural Time Differences (ITD): The tiny delay in sound reaching one ear before the other.
  2. Interaural Level Differences (ILD): The difference in volume between ears caused by the "shadow" of the head.

If a hearing aid or AR headset separates a voice but strips these cues, the user hears a "voice in their head" rather than a person standing to their left. Furthermore, existing binaural systems often suffer from high latency, making them unusable for real-time interaction.

Methods

The authors build upon TasNet (Time-domain Audio Separation Network), a high-performing model that works directly on raw waveforms rather than frequency-domain spectrograms. Their novel MIMO TasNet introduces two key architectural changes:

  • Parallel Encoders: Instead of processing channels independently, the model uses a primary encoder for the target channel and a secondary encoder for the opposite channel. This allows the network to jointly extract cross-channel spatial features.
  • Mask-and-Sum Mechanism: Traditional models estimate one "mask" to filter out noise. MIMO TasNet estimates two masks per speaker per channel. It applies one mask to the primary channel's features and another to the secondary channel's features, then sums them. This effectively performs a learned version of spatial filtering (similar to beamforming) directly in the time domain.

Crucially, the authors found that the standard training objective for speech separation (SI-SDR) actually hurts spatial preservation because it is "scale-invariant"—it ignores the volume differences (ILD) between ears. They instead use a plain Signal-to-Noise Ratio (SNR) loss to force the model to respect relative power levels.

Datasets

The researchers evaluated the model using several spatialized versions of the standard WSJ0-2mix and 3mix datasets:

  • CIPIC HRTF Database: Used to simulate anechoic (echo-free) 3D audio by convolving clean speech with Head-Related Transfer Functions from 27 different subjects.
  • DEMAND Dataset: Used to add realistic environmental noise (offices, parks, subways).
  • BRIR Sim Set: Used to simulate "echoic" environments with varying reverberation times (T60).

Key Insights and Ideas

  • Spatial Info Aids Separation: The model performs significantly better when speakers are further apart. The spatial cues aren't just something to "preserve"—the model actively uses them to distinguish between voices.
  • The Latency Breakthrough: By using small windows in the time domain, the system achieves a minimum latency of less than 5ms. This is crucial because delays over 10-20ms are perceivable and can cause "precedence effect" issues, where the brain gets confused by the timing of the original and processed sound.
  • MIMO vs. Feature Concatenation: Simply "feeding" spatial features (like phase differences) into a standard model isn't enough. The "mask-and-sum" MIMO architecture is fundamentally better at utilizing cross-channel information for both separation and cue preservation.

Key Results

  • Separation Quality: MIMO TasNet achieved an average SNR improvement (SNRi) of 15.6 dB, significantly outperforming single-channel baselines (~10.2 dB).
  • Spatial Accuracy: The model maintained an average ITD error of only 1.8 μs and an ILD error of 0.19 dB, meaning the perceived location of the speaker remains virtually unchanged.
  • Robustness: The system remained effective in noisy environments and with up to three simultaneous speakers, though performance naturally dipped compared to the two-speaker clean case.

Limitations and Future Work

  • Reverberation: While the model handles moderate echoes, performance degrades in highly reverberant rooms. The "smearing" caused by echoes makes it harder for the model to predict masks and estimate timing differences.
  • Microphone Count: The current study focuses on binaural (2-mic) setups. Future work could extend this to devices with more microphones (like smart glasses) to further improve spatial filtering.
  • Dynamic Scenes: The paper primarily tests static speaker locations; adapting to moving speakers in real-time remains a challenge for future iterations.

Summary updated

Abstract

Deep learning speech separation algorithms have achieved great success in improving the quality and intelligibility of separated speech from mixed audio. Most previous methods focused on generating a single-channel output for each of the target speakers, hence discarding the spatial cues needed for the localization of sound sources in space. However, preserving the spatial information is important in many applications that aim to accurately render the acoustic scene such as in hearing aids and augmented reality (AR). Here, we propose a speech separation algorithm that preserves the interaural cues of separated sound sources and can be implemented with low latency and high fidelity, therefore enabling a real-time modification of the acoustic scene. Based on the time-domain audio separation network (TasNet), a single-channel time-domain speech separation system that can be implemented in real-time, we propose a multi-input-multi-output (MIMO) end-to-end extension of TasNet that takes binaural mixed audio as input and simultaneously separates target speakers in both channels. Experimental results show that the proposed end-to-end MIMO system is able to significantly improve the separation performance and keep the perceived location of the modified sources intact in various acoustic scenes.
BibTeX
@inproceedings{han2020real,
  author = {Cong Han and Cong Han and Yi Luo and Nima Mesgarani},
  title = {{Real-Time Binaural Speech Separation with Preserved Spatial Cues}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2020},
  doi = {10.1109/icassp40776.2020.9053215},
}

Loading related content...

Figure