REAL-M: Towards Speech Separation on Real Mixtures

Cem Subakan, Mirco Ravanelli, Samuele Cornell, François Grondin

Abstract

Deep learning models for speech separation have achieved remarkable performance on synthetic benchmarks, yet their efficacy in real-world scenarios remains largely unquantified due to the lack of ground-truth clean audio. This paper introduces REAL-M, the first crowd-sourced dataset of "in-the-wild" speech mixtures, and a novel blind SI-SNR estimator designed to evaluate separation quality without reference signals. Our analysis reveals a substantial "reality gap": models reaching 20+ dB on synthetic data plummet to approximately 3 dB on real-world recordings. Despite this drop, relative model rankings remain consistent, validating current research directions while highlighting the need for more realistic evaluation frameworks.

Introduction

The field of speech separation has seen rapid progress, with state-of-the-art models unmixing voices with high precision on synthetic datasets like WSJ0-2Mix. These datasets are typically created by mathematically summing clean recordings, a process that fails to capture the complexities of real-world audio, such as varying microphone characteristics, room acoustics, and ambient noise.

The primary obstacle to using real-world data for evaluation is the absence of ground-truth "clean" signals for the individual speakers. Standard metrics, such as Scale-Invariant Signal-to-Noise Ratio (SI-SNR), require these references. Consequently, researchers have been confined to evaluating models on artificial data, potentially over-optimizing for laboratory conditions.

Methodology

To bridge this gap, we propose a two-pronged approach involving a new dataset and a reference-free evaluation metric.

The REAL-M Dataset

We collected the REAL-M dataset using Amazon Mechanical Turk, gathering recordings of two people speaking simultaneously in diverse environments using various devices (e.g., smartphones, laptops). The dataset comprises over 1,400 mixtures from 50 speakers, totaling nearly three hours of audio. It includes challenging "hybrid" mixtures where participants interacted via videoconferencing software, introducing additional digital artifacts.

Blind SI-SNR Estimation

We developed a convolutional neural network (CNN) to "blindly" predict SI-SNR scores. The estimator takes the original mixture and the separated output as input, learning to estimate the quality without the clean reference.

Overview of the SI-SNR estimation framework, contrasting the oracle calculation with the proposed blind estimator.

To ensure robustness, the estimator was trained using a "pool" strategy. Instead of learning from a single separation model, it was exposed to outputs from nine different pretrained separators (including SepFormer, DPRNN, and ConvTasNet) at various stages of training. This prevents the estimator from becoming biased toward the specific artifacts of a single architecture.

Results and Analysis

Our experiments demonstrate that the blind SI-SNR estimator is a reliable proxy for traditional metrics, achieving a Pearson correlation of 0.82 to 0.87 with oracle values on synthetic data. Furthermore, user studies indicate that the estimator's scores align closely with human subjective ratings of separation quality.

The Reality Gap

The most striking finding is the performance disparity between synthetic and real-world data. While a model like SepFormer achieves ~20 dB SI-SNR on clean synthetic data, its performance drops to ~2.8 dB on the REAL-M dataset.

Comparison of model performance on the REAL-M dataset (estimated) versus the synthetic WHAMR! dataset (true) across training epochs.

As shown in the figure above, while absolute performance is significantly lower on REAL-M, the relative ranking of models (SepFormer > DPRNN > ConvTasNet) remains consistent across both synthetic (WHAMR!) and real-world datasets. This suggests that improvements on synthetic benchmarks do translate to real-world gains, even if the magnitude of those gains is smaller than previously thought.

Validation via ASR and Human Judgment

We further validated these results using Word Error Rate (WER) from a Wav2Vec 2.0 automatic speech recognition (ASR) system. On REAL-M, SepFormer achieved a WER of 60.7%, which, while high, significantly outperformed other models, mirroring the rankings provided by our SI-SNR estimator.

Conclusion and Future Work

REAL-M and the blind SI-SNR estimator provide the community with essential tools to move beyond synthetic benchmarks. Our findings highlight a significant "reality gap" that must be addressed to achieve truly robust speech separation.

Future work will involve extending these tools to multi-speaker scenarios and exploring the use of the blind estimator as a loss function for fine-tuning models directly on real-world data. The REAL-M dataset and pretrained estimator have been made publicly available to facilitate further research.

Summary updated

BibTeX
@inproceedings{subakan2022real,
  author = {Cem Subakan and Mirco Ravanelli and Samuele Cornell and François Grondin},
  title = {{REAL-M: Towards Speech Separation on Real Mixtures}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2022},
  doi = {10.1109/icassp43922.2022.9746662},
}
Figure