MaD TwinNet: Masker-Denoiser Architecture with Twin Networks for Monaural Sound Source Separation

Konstantinos Drossos, Stylianos Ioannis Mimilakis, Dmitriy Serdyuk, Gerald Schuller, Tuomas Virtanen, Yoshua Bengio

Introduction

Monaural sound source separation, particularly the extraction of a singing voice from a musical mixture, remains a significant challenge in audio signal processing. The overlapping spectral and temporal characteristics of vocals and instruments necessitate sophisticated modeling techniques. While Recurrent Neural Networks (RNNs) are effective for sequence modeling, they often exhibit a "recency bias," prioritizing local information over long-term musical structures like rhythm and melody. Previous attempts to incorporate global context, such as recurrent inference, were computationally prohibitive. This paper presents MaD TwinNet, a framework that leverages "Twin Network" regularization to achieve state-of-the-art performance with high efficiency.

Methodology

The MaD TwinNet architecture consists of two main stages: a Masker-Denoiser (MaD) pipeline and a Twin Network regularization mechanism.

Masker-Denoiser Framework

The process begins with the Masker, which takes a magnitude spectrogram of the mixture and predicts a time-frequency mask. Applying this mask to the input produces an initial vocal estimate. This estimate is then processed by the Denoiser, a Denoising Autoencoder (DAE) that filters out remaining instrumental artifacts and reconstruction noise.

Overview of the MaD TwinNet architecture, illustrating the Masker, TwinNet, and Denoiser modules.

Twin Network Regularization

The core innovation is the TwinNet regularization. During training, a "twin" RNN processes the audio sequence in reverse. The model's loss function includes a penalty term that encourages the hidden states of the forward-running RNN to align with those of the backward-running twin. This effectively allows the forward network to "look ahead" and incorporate future context into its current state. Crucially, the backward twin is only required during training, allowing for efficient, real-time-capable inference.

Experimental Results

The proposed model was trained on 116 musical mixtures from the DSD100 and MedleydB datasets. Evaluation was performed on the DSD100 test set using standard metrics: Signal-to-Distortion Ratio (SDR) and Signal-to-Interference Ratio (SIR).

Quantitative comparison of MaD TwinNet with existing state-of-the-art methods.

As shown in the results, MaD TwinNet achieved an SDR of 4.57 dB and an SIR of 8.17 dB, representing a significant improvement over previous state-of-the-art methods like MIM-RINF. Furthermore, qualitative analysis indicated that MaD TwinNet produces more consistent separation across different musical genres.

Conclusion

MaD TwinNet demonstrates that incorporating future-aware regularization via Twin Networks significantly enhances monaural source separation. While the current magnitude-based approach is highly effective, future work will focus on end-to-end waveform modeling to address phase reconstruction challenges and further improve audio quality.

Summary updated

Abstract

Monaural singing voice separation task focuses on the prediction of the singing voice from a single channel music mixture signal. Current state of the art (SOTA) results in monaural singing voice separation are obtained with deep learning based methods. In this work we present a novel deep learning based method that learns long-term temporal patterns and structures of a musical piece. We build upon the recently proposed Masker-Denoiser (MaD) architecture and we enhance it with the Twin Networks, a technique to regularize a recurrent generative network using a backward running copy of the network. We evaluate our method using the Demixing Secret Dataset and we obtain an increment to signal-to-distortion ratio (SDR) of 0.37 dB and to signal-to-interference ratio (SIR) of 0.23 dB, compared to previous SOTA results.
BibTeX
@inproceedings{drossos2018mad,
  author = {Konstantinos Drossos and Stylianos Ioannis Mimilakis and Dmitriy Serdyuk and Gerald Schuller and Tuomas Virtanen and Yoshua Bengio},
  title = {{MaD TwinNet: Masker-Denoiser Architecture with Twin Networks for Monaural Sound Source Separation}},
  booktitle = {IJCNN},
  year = {2018},
  doi = {10.1109/ijcnn.2018.8489565},
}
Figure