A Novel HMM Approach to Melody Spotting in Raw Audio Recordings

Aggelos Pikrakis, Sergios TheodoridisISMIR 2005 · OpenReview

Abstract

This paper presents a robust system for "melody spotting"—the identification of specific musical themes within raw audio recordings—using Variable Duration Hidden Markov Models (VDHMMs). By employing an enhanced Viterbi algorithm designed to account for pitch-tracking errors and performer improvisation, the system successfully identifies melodies in both solo performances and complex multi-instrument ensembles. The approach addresses the inherent challenges of raw audio search, including pitch tracking inaccuracies and performance variability, demonstrating high precision across diverse musical genres.

Introduction

While many "Query-by-Humming" systems rely on symbolic MIDI databases, the vast majority of musical data exists as raw audio (e.g., MP3, WAV). Searching these recordings for specific melodies is significantly more challenging due to three primary factors: imperfect pitch tracking, performance variability (tempo and stylistic flourishes), and the "lead" problem, where the melody is obscured by accompaniment.

Quantized fundamental frequency tracking showing typical errors and transients in raw audio.

As shown in Figure 3, even quantized pitch tracking often produces "ghost" notes or split notes (circled), which can derail standard pattern-matching algorithms. The authors propose a system that treats these errors as expected probabilistic events rather than failures.

Methodology

The proposed pipeline transforms raw waveforms into a probabilistic match through several key stages:

Feature Extraction and Transposition Invariance

The system utilizes Tolonen’s multi-pitch tracking model to extract fundamental frequencies. These are converted into musical intervals and durations. By focusing on the distance between notes rather than absolute pitch, the system achieves transposition invariance, allowing it to identify a melody regardless of the key in which it is performed.

Variable Duration HMM (VDHMM)

Standard HMMs often struggle to model the temporal duration of states. The authors employ a VDHMM that explicitly models note duration using Gaussian probability distributions. This allows the model to accommodate the natural tempo fluctuations and rhythmic "stretching" common in human performances.

State sequence in the VDHMM representing the progression of musical notes.

Figure 1 illustrates the state sequence where $Z$ represents the steady-state note and $S$ represents transitions or silences. This structure allows the model to map the symbolic query to the continuous audio stream.

The Enhanced Viterbi Algorithm

The core innovation is an enhanced Viterbi algorithm that incorporates "symbol cancellation" and "summation" logic. This allows the system to mathematically "see through" pitch-tracking errors, such as when a single note is incorrectly split into two or when a transient noise is mistaken for a note. The model also includes flexible transitions to handle missing notes or repeating sub-patterns, which are frequent in improvisational styles.

Experimental Results

The system was evaluated on two distinct datasets representing different levels of complexity:

  1. Solo Cello: Recordings of J.S. Bach’s Six Suites for Cello performed by seven different artists. The system achieved a 95% success rate in locating melody occurrences, proving robust against varying artistic interpretations.
  2. Greek Traditional Music: Complex recordings featuring a lead clarinet with ensemble accompaniment. Despite micro-tonal elements and heavy improvisation, the system spotted 83% of the patterns.

Results indicate that the "Enhanced Viterbi" logic is essential; without it, pitch-tracking errors in ensemble recordings caused standard HMM approaches to fail almost entirely.

Conclusion and Future Work

This research demonstrates that a unified mathematical framework can handle both the rigid structure of Western Classical music and the fluid nature of folk traditions. However, limitations remain, particularly the dependency on a "prominent" lead instrument. Future work will focus on improving performance in dense polyphonic mixes and extending the system to support "Audio-to-Audio" spotting, where the query itself is a raw audio snippet rather than a symbolic MIDI representation.

Summary updated

BibTeX
@inproceedings{pikrakis2005novel,
  author = {Aggelos Pikrakis and Sergios Theodoridis},
  title = {{A Novel HMM Approach to Melody Spotting in Raw Audio Recordings}},
  booktitle = {ISMIR},
  year = {2005},
  url = {https://openreview.net/forum?id=VEHn4hNFeY},
}
Figure