Neural Network Alternatives to Convolutive Audio Models for Source Separation

Shrikant Venkataramani, Y. Cem Sübakan, Paris Smaragdis

TL;DR

This paper introduces convolutional auto-encoders (CAEs) and recurrent-convolutional auto-encoders (RCAEs) as neural network alternatives to traditional convolutive NMF for audio source separation. These models achieve significantly better separation performance, particularly CAEs, by effectively capturing temporal dependencies in audio signals.

What Problem Does This Paper Address?

Audio source separation, the task of isolating individual sound sources from a mixed audio signal, is a challenging problem. Traditional methods like Non-Negative Matrix Factorization (NMF) have been successful but struggle to capture the temporal dynamics inherent in audio. Convolutive NMF (conv-NMF) improved upon this by using spectro-temporal basis elements, but implementing these models with neural networks offers greater flexibility. This paper addresses the need for more powerful and flexible neural network architectures that can learn these spectro-temporal patterns effectively for improved source separation.

Methods

The core contribution is the development of two neural network architectures:

  • Convolutional Auto-Encoder (CAE): This model is presented as a neural network equivalent to conv-NMF. It uses convolutional layers in both the encoder and decoder to learn spectro-temporal basis elements. The encoder learns to extract a latent representation (akin to activations in NMF), and the decoder reconstructs the input spectrogram using these learned bases. A key aspect is the use of a non-linearity (soft-plus function) to ensure non-negativity of activations and reconstructions, mimicking NMF's constraints.
  • Recurrent-Convolutional Auto-Encoder (RCAE): This architecture extends the CAE by replacing the convolutional encoder with a recurrent neural network (RNN) encoder. The motivation is that RNNs can theoretically capture arbitrarily long temporal dependencies, potentially offering an advantage over finite-length convolutional filters. The paper notes that while they used LSTMs in experiments, other RNN architectures could also be employed.

For the separation task itself, a novel scheme is proposed that utilizes the entire trained auto-encoder. Instead of just using the learned latent representations, the auto-encoder is trained to reconstruct the mixture spectrogram as a sum of the reconstructed source spectrograms.

Datasets

  • TIMIT Corpus: Used for training and evaluating the models. The authors describe a specific data preparation process involving randomly pairing male and female speakers, mixing their utterances at 0 dB to create test mixtures, and using the remaining utterances for training source models.

Key Insights and Ideas

  • Neural Networks as Flexible NMF: The paper beautifully frames NMF and its variants as interpretable neural networks, opening the door to leveraging the power and flexibility of deep learning architectures.
  • Capturing Temporal Dependencies is Crucial: The success of convolutional and recurrent architectures highlights the importance of modeling the time-varying nature of audio signals for effective source separation.
  • CAEs Outperform Feed-Forward Equivalents: The experimental results strongly suggest that convolutional architectures (CAE) offer a significant advantage over simpler feed-forward auto-encoders for this task.
  • RCAEs Show Promise, but CAEs Lead for Speech: While RCAEs demonstrate the potential of incorporating recurrent components, CAEs achieved superior results for speech separation in this study, suggesting that the specific temporal patterns in speech might be well-suited to convolutional filters.
  • Generative Training for Robustness: The paper emphasizes that their generative training approach (learning from clean sources) makes the models more robust to different types of mixtures compared to discriminative training methods.

Key Results

  • Significant Improvement with CAEs: The convolutional auto-encoder (CAE) models significantly outperformed their feed-forward counterparts in terms of Signal-to-Distortion Ratio (SDR) and Signal-to-Interference Ratio (SIR).
  • Reduced Interference: The improvement in SIR for CAEs was particularly notable, indicating that these models are better at suppressing unwanted sources.
  • RCAE Performance: RCAE models also showed improvement over feed-forward models, though not as pronounced as CAEs for speech mixtures.
  • Optimal Basis Number: For both CAE and RCAE, performance peaked around 80 bases (decoder filters), with performance degrading less significantly for other values compared to feed-forward models.

Limitations and Future Work

  • RCAE Performance Gap: While RCAEs showed promise, they did not outperform CAEs for speech separation. The authors suggest this experimentation serves as a "proof of concept" for exploring non-uniform auto-encoder architectures, implying further work could bridge this gap.
  • Generalizability: The paper mentions that their generative training approach makes models less restricted to specific mixture types, but further exploration on diverse datasets and applications is implied.
  • Further Extensions: The authors explicitly state that their approach allows for "several extensions and generalizations," hinting at future work exploring different auto-encoder architectures and training strategies.

Summary updated

Abstract

Convolutive Non-Negative Matrix Factorization model factorizes a given audio spectrogram using frequency templates with a temporal dimension. In this paper, we present a convolutional auto-encoder model that acts as a neural network alternative to convolutive NMF. Using the modeling flexibility granted by neural networks, we also explore the idea of using a Recurrent Neural Network in the encoder. Experimental results on speech mixtures from TIMIT dataset indicate that the convolutive architecture provides a significant improvement in separation performance in terms of BSSeval metrics.
BibTeX
@inproceedings{venkataramani2017neural,
  author = {Shrikant Venkataramani and Y. Cem Sübakan and Paris Smaragdis},
  title = {{Neural Network Alternatives to Convolutive Audio Models for Source Separation}},
  booktitle = {MLSP},
  year = {2017},
  url = {https://openreview.net/forum?id=pF9fX2kWFr},
}
Figure