Neural network alternatives toconvolutive audio models for source separation

Shrikant Venkataramani, Y. Cem Sübakan, Paris Smaragdis

TL;DR

This paper introduces novel neural network architectures, specifically Convolutional Auto-Encoders (CAEs) and Recurrent-Convolutional Auto-Encoders (RCAEs), as alternatives to traditional Convolutive Non-Negative Matrix Factorization (conv-NMF) for audio source separation. These deep learning models demonstrate significant improvements in separation performance compared to feed-forward neural network approaches, particularly in handling temporal dependencies within audio signals.

What Problem Does This Paper Address?

Audio source separation, the task of isolating individual sound sources from a mixed audio signal, is a challenging problem. Traditional methods like Non-Negative Matrix Factorization (NMF) have been effective but struggle to capture the temporal dynamics inherent in audio. Convolutive NMF (conv-NMF) was developed to address this by using spectro-temporal basis elements, but it still relies on a fixed factorization model. This paper aims to bridge the gap between the modeling power of neural networks and the effectiveness of spectro-temporal modeling for source separation, proposing adaptive neural network architectures that can learn these complex patterns automatically.

Methods

The core contribution is the development of two neural network architectures:

  • Convolutional Auto-Encoder (CAE): This model acts as a direct neural network equivalent to conv-NMF. It uses convolutional layers for both encoding (extracting features) and decoding (reconstructing the input). The architecture is designed to learn spectro-temporal basis elements, similar to conv-NMF, but with the flexibility of neural networks. Non-negativity constraints are enforced through activation functions.
  • Recurrent-Convolutional Auto-Encoder (RCAE): This is an extension of the CAE that incorporates Recurrent Neural Networks (RNNs) in the encoder. The motivation is that RNNs can theoretically capture arbitrarily long temporal dependencies, potentially offering an advantage over finite-length convolutional filters. The paper explores using LSTM architectures for the recurrent components.

For the separation task itself, the authors propose a novel scheme that utilizes the entire trained auto-encoder for separation, rather than just the latent representation. Given a mixture spectrogram $X_m$, they aim to find source spectrograms $X_1$ and $X_2$ such that $X_m \approx \text{Ae}(X_1/\theta_1) + \text{Ae}(X_2/\theta_2)$, where $\text{Ae}$ represents the auto-encoder and $\theta$ its learned parameters. This approach is trained by minimizing the KL divergence between the mixture and the sum of the reconstructed sources.

Datasets

  • TIMIT Corpus: Used for training and evaluating the models. The authors describe a specific data preparation process involving randomly selecting male-female speaker pairs, mixing utterances at 0 dB, and using remaining utterances for training.

Key Insights and Ideas

  • Neural Networks as Conv-NMF Equivalents: The paper successfully demonstrates that convolutional auto-encoders can serve as flexible, data-driven alternatives to fixed conv-NMF models, learning spectro-temporal basis elements directly from data.
  • RNNs for Enhanced Temporal Modeling: Incorporating RNNs into the encoder (RCAE) offers a promising direction for capturing longer temporal dependencies, though it didn't outperform CAEs on the tested speech mixtures.
  • End-to-End Separation with Auto-encoders: The proposed separation scheme, which uses the full auto-encoder architecture to directly estimate source spectrograms, is a novel and generalized approach that can be applied to various auto-encoder architectures.
  • Convolutive Models Outperform Feed-Forward: The experimental results strongly suggest that spectro-temporal modeling (via CAEs and RCAEs) is crucial for achieving superior source separation performance compared to feed-forward auto-encoder models that don't explicitly capture these dynamics.

Key Results

  • Significant Performance Improvement: The CAE models significantly outperformed their feed-forward counterparts in terms of median BSS_eval metrics (SDR, SIR, SAR).
  • Reduced Interference: The improvement in SDR was largely attributed to a significant reduction in interference, as evidenced by the SIR plots.
  • Robustness to Number of Bases: The performance of the convolutive models was relatively robust to the number of bases (K), with a peak around K=80 but minimal degradation for other values. The variance in SDR was also observed to be lower for higher K.
  • RCAE Performance: While RCAE models also outperformed FF models, their improvement was less pronounced than that of CAEs for speech mixtures.

Limitations and Future Work

  • RCAE vs. CAE for Speech: While RCAE models showed promise, they did not outperform CAE models for speech mixture separation in this study. The authors suggest this experimentation serves as a "definite proof of concept" for exploring non-uniform auto-encoder architectures.
  • Generalizability: The paper focuses on speech and audio signals. Future work could explore the application of these models to other types of audio data or even different signal processing domains.
  • Phase Information: The current approach primarily works with magnitude spectrograms. Incorporating phase information could lead to further improvements.
  • Discriminative vs. Generative Training: The paper focuses on generative training of auto-encoders. Exploring discriminative training approaches for these architectures could be a future direction.

Summary updated

Abstract

Convolutive Non-Negative Matrix Factorization model factorizes a given audio spectrogram using frequency templates with a temporal dimension. In this paper, we present a convolutional auto-encoder model that acts as a neural network alternative to convolutive NMF. Using the modeling flexibility granted by neural networks, we also explore the idea of using a Recurrent Neural Network in the encoder. Experimental results on speech mixtures from TIMIT dataset indicate that the convolutive architecture provides a significant improvement in separation performance in terms of BSS eval metrics.
BibTeX
@inproceedings{venkataramani2017neural,
  author = {Shrikant Venkataramani and Y. Cem Sübakan and Paris Smaragdis},
  title = {{Neural network alternatives toconvolutive audio models for source separation}},
  booktitle = {MLSP},
  year = {2017},
  url = {https://openreview.net/forum?id=pF9fX2kWFr},
}
Figure