Abstract
Singing melody extraction (SME) from polyphonic music is a critical task in music information retrieval, yet it faces significant challenges due to the scarcity of high-quality labeled data and the "genre gap" in model generalization. We propose MCSSME, a novel semi-supervised framework that leverages multi-task contrastive learning and self-consistency regularization. By enforcing consistency across multiple network resolutions and utilizing auxiliary tasks such as transformation prediction and domain classification, MCSSME learns robust, genre-agnostic representations. Our evaluation across four benchmark datasets shows that MCSSME achieves state-of-the-art performance, surpassing existing supervised models even with limited labeled training data.
Introduction
Singing melody extraction (SME) involves isolating the fundamental frequency ($f_0$) of the lead vocal from complex polyphonic audio. Despite the success of deep learning, two major obstacles remain:
- Data Scarcity: Pixel-level labeling of spectrograms is labor-intensive and requires expert musical knowledge, creating a bottleneck in available training data.
- The Genre Gap: Models often fail to generalize across different musical genres (e.g., pop vs. classical) because genre-specific acoustic characteristics act as noise that confuses pitch tracking.
Traditional semi-supervised approaches, such as the "Mean Teacher" paradigm, often struggle with SME because the task is highly sensitive to input disturbances; even minor spectrogram changes can lead to significant drops in pitch accuracy.
Methodology
The MCSSME framework addresses these challenges through a multi-faceted architectural approach centered on a pyramid-shaped feature extractor.
The architecture incorporates four primary components designed to maximize the utility of unlabeled data:
- Self-Consistency Regularization (SCR): Instead of relying solely on final outputs, the model generates predictions at multiple network layers (resolutions). KL-divergence is employed to ensure these multi-level predictions remain consistent, allowing the model to "self-correct" on unlabeled data.
- Multi-Task Learning (MTL): Beyond melody extraction, the network learns to identify audio transformations (e.g., filtering, noise addition). This auxiliary task forces the model to capture more robust, general audio features.
- Contrastive Embedding Learning: Inspired by SimCLR, this module maps original and transformed audio segments close together in feature space while distancing dissimilar segments, strengthening the internal representation of melodic structures.
- Domain Classification: To bridge the genre gap, an adversarial domain classifier forces the model to map data from diverse genres into a shared, genre-agnostic subspace.
Key Insights and Results
A core innovation of MCSSME is treating genre variation as a domain adaptation problem. By employing domain classification, the model effectively ignores genre-specific "noise" to focus on the underlying melody. Furthermore, the use of SCR across a pyramid-shaped extractor provides greater resilience compared to traditional teacher-student models, which are prone to noisy pseudo-labels.
Experimental results confirm the effectiveness of this approach. MCSSME outperformed six state-of-the-art methods across benchmarks including ADC2004, MIREX 05, iKala, and MedleyDB. Notably, on the ADC2004 dataset, it achieved an Overall Accuracy (OA) of 76.7%, a significant improvement over both the previous best semi-supervised models and fully supervised alternatives like FTANet. Ablation studies further demonstrated that each component—SCR, MTL, and domain classification—is essential for achieving these gains.
Conclusion
MCSSME provides a robust solution for semi-supervised singing melody extraction, effectively tackling data scarcity and genre generalization. While the model shows superior performance, future work will focus on reducing melody detection errors in non-vocal segments by refining signal transformation techniques and potentially integrating dedicated voicing detection modules.
Summary updated