Abstract
Stacked dilated convolutions used in Wavenet have been shown effective for generating high-quality audios. By replacing pooling/striding with dilation in convolution layers, they can preserve high-resolution information and still reach distant locations. Producing high-resolution predictions is also crucial in music source separation, whose goal is to separate different sound sources while maintain the quality of the separated sounds. Therefore, in this paper, we use stacked dilated convolutions as the backbone for music source separation. Although stacked dilated convolutions can reach wider context than standard convolutions do, their effective receptive fields are still fixed and might not be wide enough for complex music audio signals. To reach even further information at remote locations, we propose to combine a dilated convolution with a modified GRU called Dilated GRU to form a block. A Dilated GRU receives information from k-step before instead of the previous step for a fixed k. This modification allows a GRU unit to reach a location with fewer recurrent steps and run faster because it can execute in parallel partially. We show that the proposed model with a stack of such blocks performs equally well or better than the state-of-the-art for separating both vocals and accompaniment.
Introduction
Music source separation, often referred to as the "cocktail party problem" for audio, involves isolating individual tracks—such as vocals, drums, or bass—from a single mixture. This task presents two primary challenges: maintaining high audio resolution and capturing long-range temporal context. Traditional architectures like U-Nets often rely on downsampling to manage computational costs, which can sacrifice fine-grained detail. Conversely, standard Recurrent Neural Networks (RNNs) can capture temporal dependencies but are inherently slow due to their sequential nature. This paper proposes a solution that maintains full temporal resolution while efficiently capturing distant context through a novel combination of dilated convolutions and a modified "Dilated GRU."
Methods
The core of the proposed architecture is the Dilated Recurrent-Dilated Convolution (D2) block. This block integrates two key components designed for high-resolution, long-context processing:
Dilated GRU
In a standard Gated Recurrent Unit (GRU), the state at time $t$ depends on the state at $t-1$. The proposed Dilated GRU modifies this such that the state at $t$ depends on the state at $t-k$, where $k$ is the dilation factor. This allows the model to "skip" across time, reaching distant context with fewer recurrent steps. A significant advantage of this approach is parallelization: the sequence can be partitioned into $k$ disjoint sets that can be processed simultaneously, significantly improving speed.
Dilated Grouped Convolutions
Stacked on top of the GRU, these layers use dilated kernels to expand the receptive field without increasing parameter count. Grouped convolutions are employed to further reduce memory and computational overhead.
The full model consists of a stack of these D2 blocks. Unlike encoder-decoder structures, this model maintains constant temporal resolution throughout, avoiding the artifacts typically introduced by upsampling layers.
Results and Discussion
The model was evaluated on the MUSDB18 dataset, comprising 100 training and 50 evaluation songs, each split into vocals, drums, bass, and "other" tracks.
- Performance: The proposed model achieved state-of-the-art results in Signal-to-Distortion Ratio (SDR), showing a 0.25 dB improvement for vocals and a 0.57 dB improvement for accompaniment over the previous leading TAK1 model.
- Efficiency: The Dilated GRU (with dilation $k=2$) reduced processing time by approximately 20% compared to a standard GRU, with negligible impact on separation quality.
- Hierarchical Processing: An ablation study revealed a hierarchical "cleaning" process, where lower blocks in the stack capture simpler background noise while deeper blocks refine complex signals like vocals.
Conclusion
This work demonstrates that high-resolution music source separation can be achieved efficiently by combining dilated convolutions with Dilated GRUs. By avoiding downsampling and enabling parallel recurrence, the model reaches wide temporal contexts without sacrificing speed or detail. Future work could address the "capacity competition" between different sources by training specialized networks for specific instruments and exploring more complex musical genres like EDM or symphonic arrangements.
Summary updated