Vocal Pitch Extraction in Polyphonic Music Using Convolutional Residual Network

Mingye Dong, Jie Wu, Jian Luan

Abstract

Extracting vocal pitch from polyphonic music is a challenging task due to the presence of loud instrumental accompaniment. This paper proposes a deep Convolutional Residual Network (ResNet) that operates directly on raw audio waveforms, bypassing traditional spectrogram-based features. By leveraging residual learning to train deeper architectures, the model achieves a significant performance boost—improving overall accuracy by approximately 5% over existing state-of-the-art methods. This demonstrates the utility of preserving phase information and using deep feature extraction in complex acoustic environments.

Introduction

Vocal pitch extraction, or fundamental frequency ($f_0$) estimation, is a foundational task in music information retrieval. In polyphonic settings, where a singer's voice is layered over loud instruments, this task becomes particularly difficult. Conventional deep learning models typically address this by converting audio into log-mel spectrograms. However, this conversion discards phase information, which contains subtle timing details helpful for distinguishing a voice from background noise. While raw waveform processing has been explored, previous attempts often relied on shallow networks that struggled with the complexity of contaminated musical data. This research bridges the gap by introducing a deep architecture capable of learning complex features directly from raw sound.

Methods: The Convolutional Residual Network (CRN)

The authors propose a Convolutional Residual Network (CRN) designed for end-to-end learning from raw audio.

The proposed CRN architecture, showing the flow from 1024 raw audio samples through eight convolutional layers with residual connections to a final 77-state classification output.

Key components of the architecture include:

  • Raw Waveform Input: The model processes 1024 raw audio samples (approximately 128ms at an 8kHz sampling rate) as its primary input.
  • Residual Learning: The network consists of eight convolutional layers organized into residual blocks. These blocks utilize "skip connections" to add a layer's input to its output, mitigating the vanishing gradient problem and enabling the training of deeper, more effective networks.
  • Classification Framework: Pitch extraction is treated as a classification task. The frequency range (87.3Hz to 784Hz) is divided into 76 mel-scale bins, with an additional bin for silence or unvoiced segments, totaling 77 possible states.
  • Label Smoothing: To help the model learn the relationship between adjacent frequencies, a smoothed target vector is used during training, distributing weight to neighboring bins.

Experimental Setup and Datasets

The study utilizes the MIR-1k dataset, a collection of 1,000 Chinese pop song snippets. The dataset provides separate tracks for vocals and accompaniment, allowing the researchers to mix them at specific Signal-to-Noise Ratios (SNRs) of 0dB (equal volume) and +5dB (vocals slightly louder) to evaluate the model's robustness. For the experiment, 900 songs were used for training and validation, while 100 were reserved for testing.

Key Results and Insights

The proposed CRN-Raw model was compared against two strong baselines: a standard shallow DNN using raw waveforms and a specialized CNN using Harmonic Constant-Q Transform (HCQT) features (DSM-HCQT).

Comparison of the CNN output against ground truth for 0dB and +5dB SNR scenarios, illustrating the model's ability to track pitch accurately despite heavy accompaniment.
  • Performance Gains: At 0dB SNR, the CRN-Raw achieved an Overall Accuracy (OA) of 78.4%, outperforming the HCQT-based model (73.3%) and the shallow DNN (74.5%).
  • Pitch Precision: The model reached a Raw Pitch Accuracy (RPA) of 78.6% (within a 50-cent tolerance), representing a 4.4% improvement over the HCQT baseline.
  • The Value of Depth and Phase: The superior performance suggests that phase information in raw waveforms—typically lost in spectrograms—is indeed useful for separating vocals. Furthermore, the ResNet architecture allows the model to automatically learn feature extractors that are more effective than hand-crafted ones like HCQT.
  • Improved Voice Activity Detection: The model significantly reduced the Voicing False Alarm (VFA) rate, demonstrating a better ability to distinguish the spectral signature of the human voice from instrumental accompaniment.

Limitations and Future Work

While the CRN-Raw shows high robustness, certain instruments like guitar or piano still pose challenges to accuracy. Additionally, the current frame-by-frame prediction can lead to minor temporal "jitter." The authors suggest that future work could incorporate temporal smoothing or dynamic programming (such as a Viterbi decoder) to refine the output sequence. Expanding the training data to include a wider variety of instruments will also be a focus for improving general robustness.

Summary updated

Abstract

Pitch extraction, also known as fundamental frequency estimation, is a long-term task in audio signal processing. Especially, due to the presence of accompaniment, vocal pitch extraction in polyphonic music is more challenging. So far, most of deep learning approaches use log mel spectrogram as input, which neglect the phase information. In addition, shallow networks have been applied on waveform directly, which may not handle contaminated vocal data well. In this paper, a deep convolutional residual network is proposed. It analyzes and extracts effective feature from waveform automatically. Residual learning can reduce model degradation due to the skip connection and residual mapping. In comparison to reported results, the proposed approach shows 5% and 4% improvement on overall accuracy(OA) and raw pitch accuracy(RPA) respectively.
BibTeX
@inproceedings{dong2019vocal,
  author = {Mingye Dong and Jie Wu and Jian Luan},
  title = {{Vocal Pitch Extraction in Polyphonic Music Using Convolutional Residual Network}},
  booktitle = {Interspeech},
  year = {2019},
  doi = {10.21437/interspeech.2019-2286},
}
Figure