Introduction
Monaural sound source separation, particularly the extraction of a singing voice from a musical mixture, remains a significant challenge in audio signal processing. The overlapping spectral and temporal characteristics of vocals and instruments necessitate sophisticated modeling techniques. While Recurrent Neural Networks (RNNs) are effective for sequence modeling, they often exhibit a "recency bias," prioritizing local information over long-term musical structures like rhythm and melody. Previous attempts to incorporate global context, such as recurrent inference, were computationally prohibitive. This paper presents MaD TwinNet, a framework that leverages "Twin Network" regularization to achieve state-of-the-art performance with high efficiency.
Methodology
The MaD TwinNet architecture consists of two main stages: a Masker-Denoiser (MaD) pipeline and a Twin Network regularization mechanism.
Masker-Denoiser Framework
The process begins with the Masker, which takes a magnitude spectrogram of the mixture and predicts a time-frequency mask. Applying this mask to the input produces an initial vocal estimate. This estimate is then processed by the Denoiser, a Denoising Autoencoder (DAE) that filters out remaining instrumental artifacts and reconstruction noise.
Twin Network Regularization
The core innovation is the TwinNet regularization. During training, a "twin" RNN processes the audio sequence in reverse. The model's loss function includes a penalty term that encourages the hidden states of the forward-running RNN to align with those of the backward-running twin. This effectively allows the forward network to "look ahead" and incorporate future context into its current state. Crucially, the backward twin is only required during training, allowing for efficient, real-time-capable inference.
Experimental Results
The proposed model was trained on 116 musical mixtures from the DSD100 and MedleydB datasets. Evaluation was performed on the DSD100 test set using standard metrics: Signal-to-Distortion Ratio (SDR) and Signal-to-Interference Ratio (SIR).
As shown in the results, MaD TwinNet achieved an SDR of 4.57 dB and an SIR of 8.17 dB, representing a significant improvement over previous state-of-the-art methods like MIM-RINF. Furthermore, qualitative analysis indicated that MaD TwinNet produces more consistent separation across different musical genres.
Conclusion
MaD TwinNet demonstrates that incorporating future-aware regularization via Twin Networks significantly enhances monaural source separation. While the current magnitude-based approach is highly effective, future work will focus on end-to-end waveform modeling to address phase reconstruction challenges and further improve audio quality.
Summary updated