DJCM: A Deep Joint Cascade Model for Singing Voice Separation and Vocal Pitch Estimation

Haojie Wei, Xueke Cao, Wenbo Xu, Tangpeng Dan, Yueguo Chen

TL;DR

This paper introduces DJCM, a novel deep learning model that simultaneously separates singing voices from music and estimates their vocal pitches. By employing a joint cascade architecture and carefully weighting the loss functions for each task, DJCM achieves state-of-the-art performance, outperforming previous methods in both singing voice separation and vocal pitch estimation.

What Problem Does This Paper Address?

Extracting clean singing voices from mixed music (Singing Voice Separation or SVS) and determining the precise pitch of those vocals (Vocal Pitch Estimation or VPE) are crucial tasks in music information retrieval. While existing methods tackle these individually or in simple combined approaches, they face limitations. Pipeline methods, which process SVS first and then VPE, suffer from a mismatch between training and testing data distributions. Naive joint learning methods, which simply sum the losses of both tasks, can lead to misaligned objectives, hindering overall performance. This paper aims to bridge this gap by proposing a more effective joint learning framework that addresses these issues.

Methods

The core of the proposed solution is the Deep Joint Cascade Model (DJCM). This model features a novel joint cascade structure that allows for concurrent training of both SVS and VPE. The architecture consists of two main modules:

  • SVS Module: This module takes the mixture music spectrogram as input and uses a deep convolutional encoder (composed of Residual Encoder Blocks - REBs) to extract features. These features are then processed by a decoder (using Residual Decoder Blocks - RDBs) to reconstruct the clean vocal spectrogram.
  • VPE Module: This module takes the predicted vocal spectrogram from the SVS module as input. It shares a similar encoder structure with the SVS module but employs a different decoder, including a bidirectional GRU and a fully connected layer, to predict vocal pitches.

A key innovation is the task-specific weighting of loss functions. Instead of simply summing the losses (e.g., Mean Absolute Error for SVS and weighted Binary Cross-Entropy for VPE), DJCM uses a hyperparameter w to dynamically adjust their contribution. This allows for better alignment of the distinct objectives of SVS and VPE, preventing one task from dominating the other and improving overall performance.

Datasets

The primary dataset used for training and evaluation is MIR-1K. This dataset is notable for containing 1000 song clips with both music accompaniment and singing voice, along with ground truth vocal pitch information in semitones. The authors mention that MIR-1K is, to their knowledge, the only dataset available that supports evaluation for both SVS and VPE tasks simultaneously. The dataset was split into 80% for training and 20% for testing.

Key Insights and Ideas

  • The power of joint learning: The results strongly suggest that jointly learning SVS and VPE is more effective than treating them as separate tasks or using naive summation of losses. The tasks appear to mutually benefit each other.
  • Cascade structure matters: The proposed cascade structure, where VPE operates on the output of SVS, seems to be more robust. This is particularly evident in how it handles unseparated accompaniment, making the VPE module more resilient.
  • Loss weighting is critical: The ablation studies on the w parameter highlight that simply adding losses is suboptimal. Carefully balancing the SVS and VPE objectives through learned weights is crucial for achieving optimal performance on both tasks.
  • SVS can improve VPE, and vice versa: The experiments show that when the model is tuned to prioritize VPE (lower w), SVS performance slightly degrades, but VPE improves. Conversely, when prioritizing SVS (higher w), SVS improves, and VPE also benefits due to the improved quality of the separated vocals.

Key Results

DJCM achieved state-of-the-art performance on the MIR-1K dataset:

  • Singing Voice Separation: An improvement of 0.45 dB in SDR (Signal-to-Distortion Ratio) compared to the best pipeline method.
  • Vocal Pitch Estimation: An improvement of 2.86% in Overall Accuracy (OA) compared to the best pipeline method.

Ablation studies also revealed:

  • DJCM outperformed other joint model structures like "Shared Bottom" and "MMOE" variants.
  • The optimal weighting w for balancing SVS and VPE was found to be around 0.6 for VPE-focused performance and higher values for SVS-focused performance, demonstrating the flexibility of the joint learning approach.

Limitations and Future Work

The authors acknowledge that the best performance on pitch transcription was achieved with a "sub-optimal vocal separation module" when w was set to 0.6. This suggests a trade-off that might be further optimized.

Future work directions include:

  • Exploring more suitable joint learning model structures.
  • Developing more effective multi-task optimization methods for joint cascade models.

Summary updated

Abstract

Singing voice separation and vocal pitch estimation are pivotal tasks in music information retrieval. Existing methods for simultaneous extraction of clean vocals and vocal pitches can be classified into two categories: pipeline methods and naive joint learning methods. However, the efficacy of these methods is limited by the following problems: On the one hand, pipeline methods train models for each task independently, resulting a mismatch between the data distributions at the training and testing time. On the other hand, naive joint learning methods simply add the losses of both tasks, possibly leading to a misalignment between the distinct objectives of each task. To solve these problems, we propose a Deep Joint Cascade Model (DJCM) for singing voice separation and vocal pitch estimation. DJCM employs a novel joint cascade model structure to concurrently train both tasks. Moreover, task-specific weights are used to align different objectives of both tasks. Experimental results show that DJCM achieves state-of-the-art performance on both tasks, with great improvements of 0.45 in terms of Signal-to-Distortion Ratio (SDR) for singing voice separation and 2.86% in terms of Overall Accuracy (OA) for vocal pitch estimation. Furthermore, extensive ablation studies validate the effectiveness of each design of our proposed model. The code of DJCM is available at https://github.com/Dream-High/DJCM .
BibTeX
@inproceedings{wei2024djcm,
  author = {Haojie Wei and Xueke Cao and Wenbo Xu and Tangpeng Dan and Yueguo Chen},
  title = {{DJCM: A Deep Joint Cascade Model for Singing Voice Separation and Vocal Pitch Estimation}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2024},
  url = {https://openreview.net/forum?id=KjeY8uLpXL},
}
Figure