TL;DR
This paper introduces DJCM, a novel deep learning model that simultaneously separates singing voices from music and estimates their vocal pitches. By employing a joint cascade architecture and carefully weighting the loss functions for each task, DJCM achieves state-of-the-art performance, outperforming previous methods in both singing voice separation and vocal pitch estimation.
What Problem Does This Paper Address?
Extracting clean singing voices from mixed music (Singing Voice Separation or SVS) and determining the precise pitch of those vocals (Vocal Pitch Estimation or VPE) are crucial tasks in music information retrieval. While existing methods tackle these individually or in simple combined approaches, they face limitations. Pipeline methods, which process SVS first and then VPE, suffer from a mismatch between training and testing data distributions. Naive joint learning methods, which simply sum the losses of both tasks, can lead to misaligned objectives, hindering overall performance. This paper aims to bridge this gap by proposing a more effective joint learning framework that addresses these issues.
Methods
The core of the proposed solution is the Deep Joint Cascade Model (DJCM). This model features a novel joint cascade structure that allows for concurrent training of both SVS and VPE. The architecture consists of two main modules:
- SVS Module: This module takes the mixture music spectrogram as input and uses a deep convolutional encoder (composed of Residual Encoder Blocks - REBs) to extract features. These features are then processed by a decoder (using Residual Decoder Blocks - RDBs) to reconstruct the clean vocal spectrogram.
- VPE Module: This module takes the predicted vocal spectrogram from the SVS module as input. It shares a similar encoder structure with the SVS module but employs a different decoder, including a bidirectional GRU and a fully connected layer, to predict vocal pitches.
A key innovation is the task-specific weighting of loss functions. Instead of simply summing the losses (e.g., Mean Absolute Error for SVS and weighted Binary Cross-Entropy for VPE), DJCM uses a hyperparameter w to dynamically adjust their contribution. This allows for better alignment of the distinct objectives of SVS and VPE, preventing one task from dominating the other and improving overall performance.
Datasets
The primary dataset used for training and evaluation is MIR-1K. This dataset is notable for containing 1000 song clips with both music accompaniment and singing voice, along with ground truth vocal pitch information in semitones. The authors mention that MIR-1K is, to their knowledge, the only dataset available that supports evaluation for both SVS and VPE tasks simultaneously. The dataset was split into 80% for training and 20% for testing.
Key Insights and Ideas
- The power of joint learning: The results strongly suggest that jointly learning SVS and VPE is more effective than treating them as separate tasks or using naive summation of losses. The tasks appear to mutually benefit each other.
- Cascade structure matters: The proposed cascade structure, where VPE operates on the output of SVS, seems to be more robust. This is particularly evident in how it handles unseparated accompaniment, making the VPE module more resilient.
- Loss weighting is critical: The ablation studies on the
wparameter highlight that simply adding losses is suboptimal. Carefully balancing the SVS and VPE objectives through learned weights is crucial for achieving optimal performance on both tasks. - SVS can improve VPE, and vice versa: The experiments show that when the model is tuned to prioritize VPE (lower
w), SVS performance slightly degrades, but VPE improves. Conversely, when prioritizing SVS (higherw), SVS improves, and VPE also benefits due to the improved quality of the separated vocals.
Key Results
DJCM achieved state-of-the-art performance on the MIR-1K dataset:
- Singing Voice Separation: An improvement of 0.45 dB in SDR (Signal-to-Distortion Ratio) compared to the best pipeline method.
- Vocal Pitch Estimation: An improvement of 2.86% in Overall Accuracy (OA) compared to the best pipeline method.
Ablation studies also revealed:
- DJCM outperformed other joint model structures like "Shared Bottom" and "MMOE" variants.
- The optimal weighting
wfor balancing SVS and VPE was found to be around 0.6 for VPE-focused performance and higher values for SVS-focused performance, demonstrating the flexibility of the joint learning approach.
Limitations and Future Work
The authors acknowledge that the best performance on pitch transcription was achieved with a "sub-optimal vocal separation module" when w was set to 0.6. This suggests a trade-off that might be further optimized.
Future work directions include:
- Exploring more suitable joint learning model structures.
- Developing more effective multi-task optimization methods for joint cascade models.
Summary updated