Melody pitch estimation based on range estimation and candidate extraction using harmonic structure model

Seokhwan Jo, Sihyun Joo, Chang D. Yoo

Introduction

In the field of Music Information Retrieval (MIR), melody extraction involves identifying the dominant pitch sequence in complex polyphonic audio. Traditional systems often struggle with "octave errors"—misidentifying a frequency as double or half the true note—and interference from low-frequency accompaniment. Most algorithms search a broad frequency range (typically 80Hz to 1280Hz), which often introduces false positives. This paper addresses these issues by introducing a harmonic structure model refined by harmonic fitness and dynamic range estimation.

Methodology

The proposed system operates in three distinct stages to ensure robust pitch detection.

1. Candidate Extraction

The algorithm first calculates the weight of various fundamental frequency ($F0$) candidates using a harmonic structure model. This model uses Gaussian distributions to represent the expected energy at the fundamental frequency and its integer multiples (harmonics) in the frequency domain. By performing an inner-product between this model and the audio's Short-Time Fourier Transform (STFT) spectrum, the system identifies potential pitches.

The Gaussian-based harmonic structure model used to identify fundamental frequency candidates by matching expected harmonic partials.

2. Harmonic Fitness Refinement

To mitigate octave errors, the authors introduce a "harmonic fitness" score. Instead of relying solely on total energy, the algorithm verifies the presence of expected harmonic partials. If a candidate pitch lacks its corresponding "family" of harmonics in the actual audio spectrum, its weight is penalized. This ensures that mathematical artifacts or interference are not mistaken for the melody.

3. Melody Range Estimation

A key innovation of this paper is the dynamic estimation of the melody's frequency range. The algorithm identifies "reliable" candidates—those that remain stable (within $\pm100$ cents) for at least 20 frames. By analyzing the distribution of these stable notes, the system calculates a tighter, clip-specific frequency range. This pruning of the search space prevents the algorithm from selecting outliers in extreme frequency registers.

Tracking of pitch candidates over time. The system must distinguish the true melody (solid line) from various competing candidates (circles, triangles, crosses).

Evaluation and Results

The algorithm was benchmarked against the ADC 2004 and MIREX 2005 datasets using Raw Pitch Accuracy (RPA) and Raw Chroma Accuracy (RCA).

  • Performance: The method achieved an RPA of 80.8% and an RCA of 86.7% on the ADC 2004 dataset.
  • Comparative Success: It significantly outperformed established baselines, such as models by Ellis (73.2% RPA) and Ryynänen (68.6% RPA).
  • Optimization: A frame length of 64ms was found to be optimal for balancing temporal resolution and frequency precision.

Conclusion

By combining a structured harmonic model with dynamic range constraints, the proposed system effectively reduces common errors in polyphonic melody extraction. While the reliance on candidate stability may pose challenges for highly virtuosic or fragmented melodies, the approach offers a significant step forward in robust pitch estimation. Future work will focus on further refining the harmonic fitness calculations to improve performance in high-noise environments.

Summary updated

BibTeX
@inproceedings{jo2010melody,
  author = {Seokhwan Jo and Sihyun Joo and Chang D. Yoo},
  title = {{Melody pitch estimation based on range estimation and candidate extraction using harmonic structure model}},
  booktitle = {Interspeech},
  year = {2010},
  doi = {10.21437/interspeech.2010-754},
}
Figure