Extracting a singing melody from polyphonic music is notoriously difficult due to overlapping background instruments. This paper introduces TONet, a plug-and-play neural network architecture that improves melody extraction by explicitly separating the prediction of a note's tone (pitch-class) and its octave. By introducing a novel input representation that groups harmonics together, TONet significantly reduces octave-jumping errors and pitch-class mistakes, boosting the accuracy of several existing baseline models.
The Challenge: Tone and Octave Errors
Singing melody extraction aims to estimate the fundamental frequency (F0) of the lead vocal from a mixed audio waveform. This is a core task in Music Information Retrieval (MIR) with applications in music recommendation and generation.
Existing deep learning methods typically feed frequency-domain representations into Convolutional Neural Networks (CNNs). However, these models frequently make two specific types of mistakes:
- Tone errors: Predicting the wrong pitch-class within the correct octave (e.g., guessing G4 instead of A4).
- Octave errors: Predicting the correct pitch-class but in the wrong octave (e.g., guessing A3 or A5 instead of A4), which manifests as jarring vertical jumps in the predicted pitch contour.
Standard models struggle because the harmonics of a single tone are spread far apart on a standard frequency axis, falling outside the local receptive fields of standard 2D-CNN kernels.
Methodology: The TONet Architecture
To bridge the gap between human perception and CNN architectures, the authors propose TONet, which consists of two main innovations: the Tone-CFP (TCFP) representation and a Tone-Octave Fusion architecture.
Tone-CFP (TCFP) Representation
The authors take a standard Combined Frequency and Periodicity (CFP) representation and rearrange the frequency bins. In TCFP, bins belonging to the same pitch-class (harmonics) are grouped adjacently. This allows standard 2D-CNNs to easily capture harmonic relationships within their small, local receptive fields.
Tone-Octave Fusion
TONet is designed as a "plug-and-play" wrapper for existing models (like MCDNN, MSNet, or FTANet):
- Encoder: A dual-stream setup processes the standard CFP and the new TCFP independently using a chosen backbone model.
- Decoder: The combined features are fed into a Transformer-based encoder that splits the task into two branches: one predicts the tone probability map, and the other predicts the octave probability map.
- Fusion: Finally, a 1D-CNN fuses the predicted tone map, octave map, and the original combined feature map to output the final frequency-bin prediction.
Experimental Results
The model was trained on MIR-1K and MedleyDB vocal tracks and tested on standard benchmarks like ADC2004 and MIREX05. The authors introduced a new metric, Raw Octave Accuracy (ROA), to specifically measure how well the model predicts the correct octave regardless of tone errors.
Key Findings
- Reduction in Errors: Adding the tone-octave fusion mechanism improved ROA and Raw Chroma Accuracy (RCA) by a massive 5% to 16% across the ADC2004 and MIREX05 test sets.
- Overall Accuracy Boost: The full TONet architecture increased Overall Accuracy (OA) by 2% to 4% over baseline models. When using MSNet as the backbone (TO-MSNet), the model achieved an impressive 82.6% OA on ADC2004 and 85.3% OA on MIREX05.
- Ablation Validation: The TCFP representation alone improved RCA by 1.5-3% in models that utilize CNN layers, proving that grouping harmonics directly aids convolutional feature extraction.
Conclusion
TONet demonstrates that better input representations and task factorization can universally improve a wide variety of existing MIR models. By hacking the CNN receptive field through TCFP and treating pitch as a 2D coordinate (pitch-class + height), the model aligns more closely with musical inductive biases. Future work will explore more robust audio features, such as learned embeddings, to improve generalization on diverse datasets like MedleyDB.
Summary updated