Dilated Convolution with Dilated GRU for Music Source Separation

Jen-Yu Liu, Yi-Hsuan Yang

Abstract

Stacked dilated convolutions, as seen in architectures like WaveNet, are highly effective for high-quality audio generation. By utilizing dilation instead of pooling or striding, these models preserve high-resolution information while expanding their receptive field. This high-resolution capability is critical for music source separation, where the goal is to isolate individual sound sources without degrading audio quality. This paper investigates a backbone of stacked dilated convolutions for music source separation. To overcome the fixed receptive field limitations of standard dilated convolutions, we propose the "Dilated GRU," a modified gated recurrent unit that receives information from $k$ steps prior. This allows the model to capture remote context with fewer recurrent steps and enables partial parallelization. Our results demonstrate that this architecture performs at or above state-of-the-art levels for separating vocals and accompaniments.

Introduction

Music source separation involves isolating individual tracks—such as vocals, drums, or bass—from a mixed audio signal. This task is computationally demanding due to the sheer length of audio sequences; a typical three-minute song can translate into nearly 8,000 spectrogram frames. Traditional approaches often rely on encoder-decoder architectures (e.g., U-Nets) that downsample data to manage long-range dependencies, which can lead to the loss of fine-grained, high-resolution details. Alternatively, standard Recurrent Neural Networks (RNNs) process sequences step-by-step, making them slow and susceptible to vanishing gradients over thousands of frames. This research seeks to process long sequences efficiently while maintaining full resolution.

Methodology: The D2 Block

The core of the proposed architecture is the D2 Block (Dilated Recurrent-Dilated Convolution), which combines stacked dilated convolutions with a novel Dilated GRU.

Stacked Dilated Convolutions

Inspired by WaveNet, dilated convolutions skip inputs at regular intervals. This allows the network to achieve a wide receptive field without reducing the spatial resolution of the data. To maintain efficiency, the authors employ grouped convolutions, leveraging the subsequent GRU layer to facilitate information exchange across channels.

The Dilated GRU

The primary innovation is the Dilated GRU. Unlike a standard GRU that processes the current step $t$ based on the immediately preceding step $t-1$, the Dilated GRU looks back $k$ steps ($t-k$).

Temporal and channel views of the D2 block, illustrating the integration of Dilated GRU and Dilated Convolution.

This modification effectively partitions the sequence into $k$ independent, parallelizable chunks. This parallelization significantly speeds up computation and shortens the path for gradient flow, mitigating the vanishing gradient problem common in long-sequence modeling.

The overall model architecture, showing the sequence of D2 Blocks between initial and final convolution layers.

Experimental Setup

The model was evaluated using the MUSDB18 dataset, a benchmark for the SiSEC 2018 music separation campaign. The dataset includes 150 songs (100 training, 50 evaluation) sampled at 44,100 Hz, with isolated stems for vocals, drums, bass, and "other" instruments.

Results and Discussion

The proposed model achieved state-of-the-art performance compared to the top entries in the SiSEC 2018 competition.

  • Separation Quality: The model achieved a Signal-to-Distortion Ratio (SDR) of 6.85 dB for vocals, surpassing the previous state-of-the-art (TAK1) score of 6.60 dB. For accompaniment, it reached 13.40 dB, compared to the 12.83 dB baseline.
  • Computational Efficiency: Implementing a Dilated GRU with a dilation factor of 2 reduced processing time by approximately 20% relative to a standard GRU, with negligible impact on separation quality.

Key Insights

  1. Parallelized Recurrence: Dilating recurrent connections allows for the parallel execution of an inherently sequential process, improving both training speed and long-range context capture.
  2. Multi-Source Training Dynamics: Training a single model for all four sources revealed that the loss function tends to prioritize louder components like vocals. While this leads to exceptional vocal separation, it can result in slightly lower performance for quieter stems like bass compared to specialized single-source models.
  3. Layer-wise Semantic Learning: Ablation studies suggest the network learns hierarchically. Shallower layers handle structurally simpler sources (drums, bass), while deeper layers are utilized for the refinement of more complex signals (vocals).

Conclusion

By combining dilated convolutions with the Dilated GRU, this architecture provides a high-resolution, computationally efficient solution for music source separation. Future work will explore applying this model to more complex acoustic environments, such as orchestral recordings or dense electronic music.

Summary updated

Abstract

Stacked dilated convolutions used in Wavenet have been shown effective for generating high-quality audios. By replacing pooling/striding with dilation in convolution layers, they can preserve high-resolution information and still reach distant locations. Producing high-resolution predictions is also crucial in music source separation, whose goal is to separate different sound sources while maintaining the quality of the separated sounds. Therefore, this paper investigates using stacked dilated convolutions as the backbone for music source separation. However, while stacked dilated convolutions can reach wider context than standard convolutions, their effective receptive fields are still fixed and may not be wide enough for complex music audio signals. To reach information at remote locations, we propose to combine dilated convolution with a modified version of gated recurrent units (GRU) called the `Dilated GRU' to form a block. A Dilated GRU unit receives information from k steps before instead of the previous step for a fixed k. This modification allows a GRU unit to reach a location with fewer recurrent steps and run faster because it can execute partially in parallel. We show that the proposed model with a stack of such blocks performs equally well or better than the state-of-the-art models for separating vocals and accompaniments.
BibTeX
@inproceedings{liu2019dilated,
  author = {Jen-Yu Liu and Yi-Hsuan Yang},
  title = {{Dilated Convolution with Dilated GRU for Music Source Separation}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2019},
  url = {https://openreview.net/forum?id=eFaJqtnk4W},
}
Figure