Tonal representations for music retrieval: from version identification to query-by-humming

Justin Salamon, Joan Serrà, Emilia GómezInt. J. Multim. Inf. Retr 2013 · DOI · OpenReview

Introduction

Music retrieval systems have traditionally relied on metadata such as artist names and song titles. However, retrieving music based solely on the audio signal remains a significant challenge, particularly when dealing with variations in performance. This research explores how "tonal" layers—specifically melody, bass line, and harmony—can be leveraged to identify different versions of the same song (version identification) and to power "query-by-humming" (QBH) systems.

The Challenge of Invariant Representation

The core difficulty in content-based music retrieval is finding a representation that is "invariant" to performance changes. Whether a song is performed by a heavy metal band, an acoustic ensemble, or hummed by an amateur, the underlying musical "DNA" must remain detectable. This paper addresses two primary tasks:

  1. Version Identification: Detecting "cover songs" where instrumentation, tempo, and style may differ drastically.
  2. Query-by-Humming: Matching a short, often imperfect, monophonic recording of a user singing or humming to a polyphonic studio track.

Methodology: A Unified Tonal Framework

The researchers propose a unified framework using three primary tonal descriptors:

  • Harmony: Represented by Harmonic Pitch Class Profiles (HPCP), which measure the intensity of the 12 semitones in the musical scale. This feature captures the overall "color" of a song.
  • Melody and Bass Line: Extracted using state-of-the-art pitch tracking. To ensure these features are robust against performance variations, the authors apply a "semitone + octave" abstraction. This process quantizes raw pitch data and collapses it into a single octave (pitch class), creating a "chroma-like" representation for melodic content.
Figure 1: The abstraction process from raw pitch (cents) to pitch class profiles for a polyphonic audio signal.
  • Sequence Matching ($Q_{max}$): To compare these sequences, the system employs a dynamic programming algorithm ($Q_{max}$) derived from nonlinear time series analysis. This allows for partial matches and handles variations in tempo and song structure.
  • Late Fusion: Machine learning classifiers (such as Random Forests and K-Star) are used to weigh the dissimilarity scores from all three descriptors, determining the likelihood that two tracks are versions of the same song.

Experimental Setup and Results

The framework was tested on a substantial collection of 2,125 songs containing 523 version sets. For the QBH task, 118 queries from 17 subjects were matched against the full polyphonic database.

Key Insights

  • Harmony as a Baseline: Harmony (HPCP) proved to be the most robust single feature for version identification, achieving a Mean Average Precision (MAP) of 0.698.
  • The Power of Fusion: While harmony is reliable, melody and bass lines provide complementary data. Fusing these features pushed classification accuracy to 88.46%. This is particularly useful in cases like a cappella versions where harmonic accompaniment is absent.
  • QBH as Version Identification: The authors demonstrate that QBH is essentially a version identification task where the "cover" is a monophonic hummed query. By applying the same abstraction and matching techniques, the system achieved a Mean Reciprocal Rank (MRR) of 0.67 for well-tuned singers, typically placing the correct song in the top two results.
Figure 2: Tonal abstraction applied to a monophonic hummed query, demonstrating the 'common language' used to bridge the gap between user input and polyphonic database tracks.

Conclusion

By creating a "common language" through pitch-class abstraction, this research successfully bridges the gap between diverse musical performances and user-generated queries. While the system is highly accurate, future work is needed to address transcription errors in complex mixes and the computational load of the $Q_{max}$ algorithm for commercial-scale databases. Additionally, improving robustness against "de-tuning"—where a user changes key mid-hum—remains a priority for refining query-by-humming technology.

Summary updated

Abstract

In this study we compare the use of different music representations for retrieving alternative performances of the same musical piece, a task commonly referred to as version identification. Given the audio signal of a song, we compute descriptors representing its melody, bass line and harmonic progression using state-of-the-art algorithms. These descriptors are then employed to retrieve different versions of the same musical piece using a dynamic programming algorithm based on nonlinear time series analysis. First, we evaluate the accuracy obtained using individual descriptors, and then we examine whether performance can be improved by combining these music representations (i.e. descriptor fusion). Our results show that whilst harmony is the most reliable music representation for version identification, the melody and bass line representations also carry useful information for this task. Furthermore, we show that by combining these tonal representations we can increase version detection accuracy. Finally, we demonstrate how the proposed version identification method can be adapted for the task of query-by-humming. We propose a melody-based retrieval approach, and demonstrate how melody representations extracted from recordings of a cappella singing can be successfully used to retrieve the original song from a collection of polyphonic audio. The current limitations of the proposed approach are discussed in the context of version identification and query-by-humming, and possible solutions and future research directions are proposed.
BibTeX
@article{salamon2013tonal,
  author = {Justin Salamon and Joan Serrà and Emilia Gómez},
  title = {{Tonal representations for music retrieval: from version identification to query-by-humming}},
  journal = {Int. J. Multim. Inf. Retr.},
  year = {2013},
  doi = {10.1007/s13735-012-0026-0},
}
Figure