Convolutional vs. Recurrent Neural Networks for Audio Source Separation

Shariq Mobin, Brian Cheung, Bruno A. OlshausenarXiv 2018 · OpenReview

Abstract

Recent work has shown that recurrent neural networks can be trained to separate individual speakers in a sound mixture with high fidelity. Here we explore convolutional neural network models as an alternative and show that they achieve state-of-the-art results with an order of magnitude fewer parameters. We also characterize and compare the robustness and ability of these different approaches to generalize under three different test conditions: longer time sequences, the addition of intermittent noise, and different datasets not seen during training. For the last condition, we create a new dataset, RealTalkLibri, to test source separation in real-world environments. We show that the acoustics of the environment have significant impact on the structure of the waveform and the overall performance of neural network models, with the convolutional model showing superior ability to generalize to new environments.

Introduction

Audio source separation, often referred to as the "cocktail party problem," involves isolating individual voices from a complex acoustic mixture. While Bi-directional Long Short-Term Memory (BLSTM) networks have recently achieved high fidelity in this task, they are computationally intensive and often suffer from dataset bias. Most models are trained on clean, digitally summed mixtures that fail to account for real-world variables like room acoustics, microphone distance, and ambient noise. This study investigates whether convolutional architectures can offer a more efficient and robust alternative to standard recurrent models.

Methodology: The Deep Attractor Framework

The researchers utilize the Deep Attractor Framework, which treats source separation as a clustering problem within a high-dimensional embedding space. The primary innovation is the replacement of the standard BLSTM with a Dilated Convolutional Neural Network (CNN).

Dilated Convolutions and Stationarity

Unlike standard convolutions, dilated filters incorporate "gaps" between pixels, enabling a massive receptive field—integrating roughly one second of audio—without sacrificing fine-grained resolution or inflating parameter counts. The authors argue that CNNs are inherently suited for audio because speech harmonics exhibit "stationarity" in the frequency dimension; overtone patterns remain consistent regardless of their position on the spectrogram. Furthermore, unlike bi-directional RNNs that require the entire future of a recording to process a single frame, this CNN utilizes a fixed-lag delay, making it a candidate for real-time streaming applications.

Separated source components illustrating the target output of the Deep Attractor Framework in the time-frequency domain.

Experimental Setup and Datasets

The models were evaluated across three tiers of increasing difficulty:

  1. WSJ0 (Wall Street Journal): The standard benchmark for clean, simulated mixtures.
  2. LibriSpeech: Used to evaluate generalization to speakers not present in the training set.
  3. RealTalkLibri (RTL): A new dataset introduced in this paper. It consists of LibriSpeech samples played through physical speakers in a real room and recorded via microphone. This captures authentic reverberations and physical distortions absent in digital simulations.

Results and Performance Analysis

The CNN model demonstrated significant efficiency, achieving state-of-the-art results with only 1.6 million parameters, compared to the 17 million parameters required by the baseline BLSTM (DANet). On the WSJ0 test set, the CNN achieved a 10.97 dB Signal-to-Distortion Ratio (SDR), outperforming the larger BLSTM model's 10.48 dB.

A comparative visualization of separation performance. The top row shows the input mixtures, while the subsequent rows display the separated spectrograms produced by the different model architectures.

Robustness and Generalization

While both architectures handled long sequences and intermittent noise well, the CNN showed superior grace when transitioning to the RealTalkLibri dataset. Although both models experienced a performance drop in real-world conditions (falling to ~3.5 dB SDR), the CNN maintained a higher performance floor. The CNN also proved to be a "stationary process," with error rates remaining stable even when processing sequences 25 times longer than those encountered during training.

Conclusion and Future Work

The study concludes that the inductive bias of convolutions is more efficient for audio processing than the memory-heavy approach of RNNs. However, a significant "reality gap" remains between simulated and real-world performance. Common failure modes in real environments include harmonic misalignment (assigning overtones to the wrong speaker) and temporal instability (speaker identity flipping mid-sentence). Future research will focus on advanced data augmentation to simulate room acoustics during training and developing regularizers to ensure speaker consistency over time.

Summary updated

Abstract

Recent work has shown that recurrent neural networks can be trained to separate individual speakers in a sound mixture with high fidelity. Here we explore convolutional neural network models as an alternative and show that they achieve state-of-the-art results with an order of magnitude fewer parameters. We also characterize and compare the robustness and ability of these different approaches to generalize under three different test conditions: longer time sequences, the addition of intermittent noise, and different datasets not seen during training. For the last condition, we create a new dataset, RealTalkLibri, to test source separation in real-world environments. We show that the acoustics of the environment have significant impact on the structure of the waveform and the overall performance of neural network models, with the convolutional model showing superior ability to generalize to new environments. The code for our study is available at https://github.com/ShariqM/source_separation.
BibTeX
@misc{mobin2018convolutional,
  author = {Shariq Mobin and Brian Cheung and Bruno A. Olshausen},
  title = {{Convolutional vs. Recurrent Neural Networks for Audio Source Separation}},
  year = {2018},
  eprint = {1803.08629},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1803.08629},
}
Figure