A Joint Network for Singing Melody Extraction from Polyphonic Music with Attention Aggregation and Self-Consistency Training

Jiabo Jing, Ying Hu, Hao Huang, Liang He, Zhijian OuInterspeech 2025

TL;DR

This paper introduces a novel "Joint Network" for singing melody extraction (SME) that combines the strengths of a traditional spectrum-based model with a powerful pre-trained audio model. By employing an attention aggregation module and a self-consistency training strategy, their method achieves state-of-the-art performance on multiple benchmark datasets, significantly reducing octave and melody detection errors.

What Problem Does This Paper Address?

Singing melody extraction (SME) is a crucial task in music information retrieval (MIR), aiming to isolate the fundamental frequency (pitch) of the lead singer from complex polyphonic music. This is challenging because the singing voice is often masked by accompanying instruments and other vocal elements. Existing approaches typically rely on either spectrograms (time-frequency representations) or raw audio waveforms. While some attempts have been made to combine these, they often struggle with effective feature fusion and achieving ideal results. Furthermore, leveraging the power of large-scale pre-trained audio models for SME has shown promise, but integrating them effectively with traditional methods remains an open area. This paper seeks to bridge this gap by proposing a unified framework that synergistically combines a custom-built spectrum-based model with a pre-trained model, enhanced by advanced fusion and training techniques.

Methods

The proposed approach centers around a "Joint Network" that integrates two main branches:

  • Spectrum-Based Model: This branch utilizes a novel architecture inspired by Res2Net. It comprises a convolutional block for initial feature extraction, followed by three cascaded "Time-Frequency Attention Res2Blocks" (TFA-Res2Blocks). These blocks incorporate time-frequency attention to mimic human hearing and extract multi-scale features. A "Selective Feature Fusion" (SFF) module then dynamically selects and fuses these features.
  • Pre-trained Model: The paper leverages the "Deep Audio-Signal Holistic Embeddings" (Dasheng) model, a powerful pre-trained encoder trained on a massive dataset of general audio and music. This model is used as a fixed feature extractor, capturing rich musical information from the raw audio. Gated recurrent units (GRUs) are then employed to model these extracted features.

The core innovation lies in how these two branches are combined:

  • Attention Aggregation Module (AAM): This module fuses the intermediate features from both the spectrum-based and pre-trained branches. It consists of:
  • Cross-Attention (CA): Multi-head self-attention mechanisms are used to allow features from one branch to attend to features from the other, enabling complementary information exchange.
  • Adaptive Decision Fusion (ADF): This component dynamically adjusts the network's reliance on each branch's output by learning a gating mechanism, allowing for more adaptive and robust decision-making.
  • Self-Consistency Training Strategy: To further improve the performance of the individual branches and the joint network, a novel training strategy is introduced. This strategy uses both hard (ground truth) and soft labels (derived from the joint network's output) to supervise the two single-branch models. This "self-distillation" process encourages each branch to extract more task-relevant information, ultimately benefiting the overall fusion process. The training loss combines binary cross-entropy (BCE) for hard labels and Kullback-Leibler (KL) divergence for soft labels.

Datasets

The paper uses the following datasets for training and evaluation:

  • Training:
    • MIR-1K dataset (1,000 Chinese pop song clips)
    • MedleyDB (35 vocal tracks)
  • Testing:
    • ADC2004 (12 clips)
    • MIREX05 (9 clips)
    • MedleyDB (12 clips)

The authors note that the training and testing datasets do not overlap.

Key Insights and Ideas

  • Synergy of Spectrum and Pre-trained Models: The paper demonstrates that combining traditional, handcrafted feature representations (spectrograms) with powerful, learned representations from large pre-trained models yields superior results compared to using either approach alone.
  • Attention for Feature Fusion: The Attention Aggregation Module (AAM), particularly the cross-attention mechanism, is crucial for effectively integrating the diverse information captured by the two branches. It allows the network to intelligently weigh and combine features.
  • Adaptive Fusion Matters: The Adaptive Decision Fusion (ADF) component highlights the importance of dynamically adjusting the contribution of each branch based on the input, rather than using a fixed fusion strategy.
  • Self-Consistency as a Training Booster: The self-consistency training strategy is a clever way to improve the quality of individual model features by leveraging the knowledge learned by the more powerful joint model. This "teaching" mechanism helps the single branches become more effective contributors.
  • Reduced Octave Errors: The visualization suggests that the proposed Joint Network is particularly effective at reducing octave errors, a common pitfall in melody extraction.

Key Results

  • State-of-the-Art Performance: The proposed Joint Network achieves the best performance across all three benchmark datasets (ADC 2004, MIREX 05, and MEDLEY DB), outperforming six state-of-the-art methods.
  • High Accuracy Scores: The Joint Network achieved overall accuracy (OA) scores of 91.6% on ADC 2004, 92.5% on MIREX 05, and 78.9% on MEDLEY DB.
  • Component Effectiveness: Ablation studies confirm the effectiveness of each proposed component: the spectrum-based model, the pre-trained model integration, the AAM (CA and ADF), and the self-consistency training strategy. Removing these components led to significant performance degradation.
  • Qualitative Improvement: Visualizations show that the Joint Network significantly reduces melody detection errors and octave errors compared to a strong baseline like FTANet.

Limitations and Future Work

  • Computational Cost: While not explicitly detailed as a limitation, integrating two branches and using attention mechanisms likely increases computational complexity compared to single-branch models.
  • Hyperparameter Tuning: The paper mentions that hyperparameters like $\lambda_1$ and $\lambda_2$ for the loss function were empirically selected, suggesting that further tuning might yield even better results.
  • Future Work: The authors suggest that their proposed framework can be extended to other MIR tasks. Further exploration of different pre-trained models and attention mechanisms could also be beneficial. The effectiveness of the spectrum-based model with fewer parameters is also noted, hinting at potential for more efficient SME models.

Summary updated

BibTeX
@inproceedings{jing2025joint,
  author = {Jiabo Jing and Ying Hu and Hao Huang and Liang He and Zhijian Ou},
  title = {{A Joint Network for Singing Melody Extraction from Polyphonic Music with Attention Aggregation and Self-Consistency Training}},
  booktitle = {Interspeech},
  year = {2025},
}
Figure