Abstract
Stacked dilated convolutions, as seen in architectures like WaveNet, are highly effective for high-quality audio generation. By utilizing dilation instead of pooling or striding, these models preserve high-resolution information while expanding their receptive field. This high-resolution capability is critical for music source separation, where the goal is to isolate individual sound sources without degrading audio quality. This paper investigates a backbone of stacked dilated convolutions for music source separation. To overcome the fixed receptive field limitations of standard dilated convolutions, we propose the "Dilated GRU," a modified gated recurrent unit that receives information from $k$ steps prior. This allows the model to capture remote context with fewer recurrent steps and enables partial parallelization. Our results demonstrate that this architecture performs at or above state-of-the-art levels for separating vocals and accompaniments.
Introduction
Music source separation involves isolating individual tracks—such as vocals, drums, or bass—from a mixed audio signal. This task is computationally demanding due to the sheer length of audio sequences; a typical three-minute song can translate into nearly 8,000 spectrogram frames. Traditional approaches often rely on encoder-decoder architectures (e.g., U-Nets) that downsample data to manage long-range dependencies, which can lead to the loss of fine-grained, high-resolution details. Alternatively, standard Recurrent Neural Networks (RNNs) process sequences step-by-step, making them slow and susceptible to vanishing gradients over thousands of frames. This research seeks to process long sequences efficiently while maintaining full resolution.
Methodology: The D2 Block
The core of the proposed architecture is the D2 Block (Dilated Recurrent-Dilated Convolution), which combines stacked dilated convolutions with a novel Dilated GRU.
Stacked Dilated Convolutions
Inspired by WaveNet, dilated convolutions skip inputs at regular intervals. This allows the network to achieve a wide receptive field without reducing the spatial resolution of the data. To maintain efficiency, the authors employ grouped convolutions, leveraging the subsequent GRU layer to facilitate information exchange across channels.
The Dilated GRU
The primary innovation is the Dilated GRU. Unlike a standard GRU that processes the current step $t$ based on the immediately preceding step $t-1$, the Dilated GRU looks back $k$ steps ($t-k$).
This modification effectively partitions the sequence into $k$ independent, parallelizable chunks. This parallelization significantly speeds up computation and shortens the path for gradient flow, mitigating the vanishing gradient problem common in long-sequence modeling.
Experimental Setup
The model was evaluated using the MUSDB18 dataset, a benchmark for the SiSEC 2018 music separation campaign. The dataset includes 150 songs (100 training, 50 evaluation) sampled at 44,100 Hz, with isolated stems for vocals, drums, bass, and "other" instruments.
Results and Discussion
The proposed model achieved state-of-the-art performance compared to the top entries in the SiSEC 2018 competition.
- Separation Quality: The model achieved a Signal-to-Distortion Ratio (SDR) of 6.85 dB for vocals, surpassing the previous state-of-the-art (TAK1) score of 6.60 dB. For accompaniment, it reached 13.40 dB, compared to the 12.83 dB baseline.
- Computational Efficiency: Implementing a Dilated GRU with a dilation factor of 2 reduced processing time by approximately 20% relative to a standard GRU, with negligible impact on separation quality.
Key Insights
- Parallelized Recurrence: Dilating recurrent connections allows for the parallel execution of an inherently sequential process, improving both training speed and long-range context capture.
- Multi-Source Training Dynamics: Training a single model for all four sources revealed that the loss function tends to prioritize louder components like vocals. While this leads to exceptional vocal separation, it can result in slightly lower performance for quieter stems like bass compared to specialized single-source models.
- Layer-wise Semantic Learning: Ablation studies suggest the network learns hierarchically. Shallower layers handle structurally simpler sources (drums, bass), while deeper layers are utilized for the refinement of more complex signals (vocals).
Conclusion
By combining dilated convolutions with the Dilated GRU, this architecture provides a high-resolution, computationally efficient solution for music source separation. Future work will explore applying this model to more complex acoustic environments, such as orchestral recordings or dense electronic music.
Summary updated