Audio-Based Musical Version Identification: Elements and challenges

Furkan Yesiler, Guillaume Doras, Rachel M. Bittner, Christopher J. Tralie, Joan Serrà

This survey paper, published in the IEEE Signal Processing Magazine, provides a comprehensive retrospective on 20 years of research into Musical Version Identification (VI)—the computational task of detecting whether two audio tracks are different renditions (covers, remixes, live performances) of the same underlying composition.

TL;DR

This paper traces the 20-year evolution of Musical Version Identification from early symbolic sequence matching to modern deep metric learning. It provides a unified framework for understanding how machines can identify the "semantic DNA" of a song even when its tempo, key, timbre, and structure are radically altered.

What Problem Does This Paper Address?

Identifying "covers" is significantly harder than standard music fingerprinting (like Shazam). While fingerprinting looks for exact acoustic matches, VI must handle extreme variations. A heavy metal cover of a folk song shares almost no "surface" acoustic properties with the original, yet humans easily recognize them as the same song.

The authors tackle the challenge of defining and modeling musical similarity across three main axes of variance:

  1. Transposition: Changes in musical key.
  2. Tempo/Timing: Changes in speed or rhythmic "feel."
  3. Structure: Adding, removing, or reordering sections (e.g., skipping a bridge or extending a solo).

The paper also advocates for the term "musical version" over "cover song," noting that the latter has narrow genre associations and a problematic history involving racial marketing strategies in the 1950s.

Methods

The paper categorizes VI systems into a pipeline of "Building Blocks":

  • Feature Extraction: Early systems used symbolic data (MIDI). This evolved into Pitch Class Profiles (PCP) or "Chroma" features, which collapse audio into 12 bins representing the semitones of the Western scale. Modern systems use Constant-Q Transforms (CQT) and deep learning to extract "embeddings"—compact mathematical vectors that represent a song’s identity.
  • Invariance Techniques:
    • Transposition: Using the Optimal Transposition Index (OTI) or 2D Fourier Transforms to make features "key-blind."
    • Tempo: Using Dynamic Time Warping (DTW) or beat-synchronous features to align tracks of different speeds.
  • Structure: Using local alignment (Smith-Waterman algorithm) or global pooling in neural networks to find matching segments regardless of where they appear in the track.
  • Similarity Estimation: The shift from knowledge-driven heuristics (calculating distances between chroma sequences) to data-driven approaches (training Siamese networks with triplet loss to "push" versions together and "pull" non-versions apart in a high-dimensional space).

Datasets

The paper summarizes the growth of benchmarks from tiny collections to massive industrial-scale datasets:

  • MIREX collection: The early standard (1,000 tracks), though private and small.
  • covers80: A popular early public dataset with 80 pairs of songs.
  • SecondHandSongs (SHS): A massive leap forward, derived from the Million Song Dataset.
  • SHS-100K: A deep-learning-ready dataset with over 100,000 tracks.
  • Da-TACOS: A recent, highly curated benchmark designed to provide clean metadata and pre-extracted features for reproducible research.

Key Insights and Ideas

  • The Accuracy-Scalability Trade-off: For a decade, the most accurate systems were "rule-based" and used $O(N^2)$ alignment algorithms (like DTW), making them too slow for millions of songs. Deep learning finally broke this bottleneck by allowing for $O(1)$ similarity lookups using vector embeddings.
  • The "Clique" Concept: The authors define versions as belonging to a "clique" (a group of all renditions of one song). This allows researchers to use graph theory to "clean up" results—if A is a version of B, and B is a version of C, then A and C must be versions of each other.
  • Beyond Pitch: While most VI research focuses on melody and harmony, the authors point out that for genres like Hip-Hop or Electronic music, pitch-based features are often useless, highlighting a major "blind spot" in current tech.

Key Results

The paper tracks the Mean Average Precision (MAP)—a standard metric for retrieval—across 20 years.

  • The "Aha" Moment: Despite datasets growing by 100x (from 1,000 to 100,000+ tracks), MAP scores have remained remarkably high (often >0.8 on specific benchmarks).
  • Deep Learning Dominance: Since 2015, data-driven systems have matched or exceeded the accuracy of hand-crafted alignment algorithms while being orders of magnitude faster.

Limitations and Future Work

  • Lyrics: Most current systems are "deaf" to lyrics. Integrating Automated Lyrics Transcription (ALT) could solve "edge cases" where the music is unrecognizable but the words remain the same.
  • Non-Western Music: Most VI research assumes a 12-tone Western scale. This fails for traditions like Indian Art Music (Ragas), where melodic motifs don't fit into Western "chroma" bins.
  • Legal vs. Musical Definitions: A "remaster" is musically identical but legally a different entity. Current AI struggles to distinguish between a creative "version" and a legal "re-release" without using text metadata.

Summary updated

Abstract

Creating novel interpretations of existing musical compositions is and has always been an essential part of musical practice. Before the advent of recorded music, listening to a piece of music mostly meant listening to a version of it, in many cases, performed by musicians other than the original composer or performer. While this practice, along with musical notation, allows compositions to remain known for many decades or even centuries, it also provides room for artistic expression.
BibTeX
@inproceedings{yesiler2021audio,
  author = {Furkan Yesiler and Guillaume Doras and Rachel M. Bittner and Christopher J. Tralie and Joan Serrà},
  title = {{Audio-Based Musical Version Identification: Elements and challenges}},
  booktitle = {IEEE Signal Process. Mag},
  year = {2021},
  url = {https://openreview.net/forum?id=8mGoGWLxR7},
}

Loading related content...

Figure