Real-time binaural speech separation with preserved spatial cues

Cong Han, Yi Luo, Nima Mesgarani

Abstract

Deep learning speech separation algorithms have achieved great success in improving the quality and intelligibility of separated speech from mixed audio. Most previous methods focused on generating a single-channel output for each of the target speakers, hence discarding the spatial cues needed for the localization of sound sources in space. However, preserving the spatial information is important in many applications that aim to accurately render the acoustic scene such as in hearing aids and augmented reality (AR). Here, we propose a speech separation algorithm that preserves the interaural cues of separated sound sources and can be implemented with low latency and high fidelity, therefore enabling a real-time modification of the acoustic scene. Based on the time-domain audio separation network (TasNet), a single-channel time-domain speech separation system that can be implemented in real-time, we propose a multi-input-multi-output (MIMO) end-to-end extension of TasNet that takes binaural mixed audio as input and simultaneously separates target speakers in both channels. Experimental results show that the proposed end-to-end MIMO system is able to significantly improve the separation performance and keep the perceived location of the modified sources intact in various acoustic scenes.

Introduction

Most deep learning speech separation models are "Multi-Input Single-Output" (MISO). They take audio from multiple microphones but produce a single, mono-channel output for each speaker. While the speech is clear, the spatial cues—the "where" of the sound—are lost. For humans, spatial hearing relies on two primary cues: Interaural Time Differences (ITD), the tiny delay in sound reaching one ear before the other, and Interaural Level Differences (ILD), the difference in volume between ears caused by the "shadow" of the head.

If a hearing aid or AR headset separates a voice but strips these cues, the user hears a "voice in their head" rather than a person standing to their left. Furthermore, existing binaural systems often suffer from high latency, making them unusable for real-time interaction. Researchers at Columbia University have developed MIMO TasNet to address these challenges.

Methods: The MIMO Architecture

The authors build upon TasNet (Time-domain Audio Separation Network), a high-performing model that works directly on raw waveforms. Their novel MIMO TasNet introduces two key architectural changes to preserve spatiality:

  • Parallel Encoders: Instead of processing channels independently, the model uses a primary encoder for the target channel and a secondary encoder for the opposite channel. This allows the network to jointly extract cross-channel spatial features.
  • Mask-and-Sum Mechanism: Traditional models estimate one "mask" to filter out noise. MIMO TasNet estimates two masks per speaker per channel. It applies one mask to the primary channel's features and another to the secondary channel's features, then sums them. This effectively performs a learned version of spatial filtering (similar to beamforming) directly in the time domain.
Architecture of the proposed MIMO TasNet system, highlighting the parallel encoders and mask-and-sum separator.

Crucially, the authors found that the standard training objective for speech separation (SI-SDR) actually hurts spatial preservation because it is "scale-invariant"—it ignores the volume differences (ILD) between ears. They instead use a plain Signal-to-Noise Ratio (SNR) loss to force the model to respect relative power levels.

Experimental Setup

The researchers evaluated the model using several spatialized versions of the standard WSJ0-2mix and 3mix datasets. They utilized the CIPIC HRTF Database to simulate anechoic 3D audio by convolving clean speech with Head-Related Transfer Functions. To test robustness, they added realistic environmental noise from the DEMAND Dataset and simulated echoic environments with varying reverberation times using the BRIR Sim Set.

Key Results

The model performs significantly better when speakers are further apart, indicating that the system actively uses spatial cues to distinguish between voices. By using small windows in the time domain, the system achieves a minimum latency of less than 5ms, which is vital for avoiding the "precedence effect" where the brain gets confused by timing differences between original and processed sound.

Quantitative performance comparison across different angular separations and model variants, showing SNRi and spatial cue errors.

As shown in the experimental results, MIMO TasNet achieved an average SNR improvement (SNRi) of 15.6 dB, significantly outperforming single-channel baselines (~10.2 dB). More importantly, the model maintained an average ITD error of only 1.8 μs and an ILD error of 0.19 dB, meaning the perceived location of the speaker remains virtually unchanged.

Conclusion and Future Work

MIMO TasNet represents a breakthrough in real-time, spatially-aware speech separation. While the system is highly effective in moderate environments, performance degrades in highly reverberant rooms where echoes "smear" timing differences. Future work may extend this to devices with more microphones (like smart glasses) and adapt the model to handle dynamic, moving speakers in real-time.

Summary updated

Abstract

Deep learning speech separation algorithms have achieved great success in improving the quality and intelligibility of separated speech from mixed audio. Most previous methods focused on generating a single-channel output for each of the target speakers, hence discarding the spatial cues needed for the localization of sound sources in space. However, preserving the spatial information is important in many applications that aim to accurately render the acoustic scene such as in hearing aids and augmented reality (AR). Here, we propose a speech separation algorithm that preserves the interaural cues of separated sound sources and can be implemented with low latency and high fidelity, therefore enabling a real-time modification of the acoustic scene. Based on the time-domain audio separation network (TasNet), a single-channel time-domain speech separation system that can be implemented in real-time, we propose a multi-input-multi-output (MIMO) end-to-end extension of TasNet that takes binaural mixed audio as input and simultaneously separates target speakers in both channels. Experimental results show that the proposed end-to-end MIMO system is able to significantly improve the separation performance and keep the perceived location of the modified sources intact in various acoustic scenes.
BibTeX
@inproceedings{han2020real,
  author = {Cong Han and Yi Luo and Nima Mesgarani},
  title = {{Real-time binaural speech separation with preserved spatial cues}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2020},
  doi = {10.1109/icassp40776.2020.9053215},
}
Figure