Accurate and Scalable Version Identification Using Musically-Motivated Embeddings

Furkan Yesiler, Joan Serrà, Emilia Gómez

TL;DR

The MOVE (Musically-motivated Version Embeddings) framework achieves state-of-the-art results in version identification (cover song detection) by mapping audio to a standardized Euclidean latent space. By combining a novel multi-channel attention mechanism with domain-specific data augmentation, MOVE provides a highly accurate system that scales to massive datasets, outperforming previous fusion-based methods by over 10% in Mean Average Precision.

What Problem Does This Paper Address?

Version Identification (VI) is the task of automatically detecting different renditions of the same musical piece—think of a jazz cover of a rock song or a live acoustic version of a studio track. Unlike "audio fingerprinting" (which identifies exact digital copies like Shazam does), VI must ignore changes in instrumentation, tempo, key, and structure while capturing the underlying musical "soul."

The core challenge in VI is the trade-off between accuracy and scalability. Traditional high-accuracy systems rely on complex alignment algorithms that are computationally expensive, making them impossible to use on million-song databases. Recent deep learning attempts have tried to create compact "embeddings" (vector representations) for songs, but they often struggle to match the performance of these slower, traditional methods.

Methods

The authors introduce MOVE, a deep convolutional neural network designed with several "musically-motivated" components:

  • Input Representation: Instead of raw audio or standard chromagrams, they use crema-PCP, a pitch class profile derived from a pre-trained chord estimation model.
  • Transposition Invariance: To handle key changes, the model uses a "circular" convolution strategy. It doubles the input's pitch dimension and uses a specific kernel size to ensure the network identifies the same musical patterns regardless of the starting key.
  • Multi-channel Adaptive Attention: Instead of simple average pooling, MOVE uses a novel temporal summarization technique. It learns to "attend" to the most important parts of a song while using a learnable "auto-pool" parameter that can pivot between average and max-pooling.
  • Standardized Latent Space: The final layer uses non-parametric batch normalization. This ensures the resulting embeddings have zero mean and unit variance, making the Euclidean distance between vectors more stable and intuitive during training.
  • VI-Specific Data Augmentation: The model is trained by intentionally distorting songs—shifting keys, stretching time, and warping the structure—to force the network to learn features that are invariant to these common "cover song" changes.

Datasets

  • Training: A private collection of 97,905 songs (divided into ~18,000 "cliques" or version groups) sourced via the SecondHandSongs API.
  • Evaluation (Da-TACOS): A benchmark set of 15,000 songs.
  • Evaluation (YouTubeCovers - YTC): A dataset of 50 cliques with 7 versions each, used to compare against previous SOTA models.

Key Insights and Ideas

  • The Power of Standardization: The authors found that "standardizing" the embedding space (via non-parametric BN) was crucial. If the magnitude of distances in the hyper-dimensional space is disproportionate to the loss function's margin, the model fails to structure the latent space effectively.
  • Attention Matters: Moving away from simple global pooling to an adaptive attention mechanism allowed the model to focus on the most "version-defining" segments of a track, significantly boosting performance.
  • Dimensionality vs. Performance: The study found a "knee" in the performance curve around an embedding size of 2,000 dimensions. While performance continues to improve up to 16,000 dimensions, the gains diminish, suggesting a sweet spot for practical applications.
  • Hard Triplet Mining: Using an "online hard triplet mining" strategy—where the model specifically looks for the most difficult examples to distinguish during each training batch—was key to reaching SOTA levels.

Key Results

MOVE significantly outperformed existing methods on both major benchmarks:

  • Da-TACOS: MOVE achieved a Mean Average Precision (MAP) of 0.507, compared to the previous best "LateFusion" method at 0.454. It also drastically improved the Mean Rank (MR1), dropping it from 177 to 40 (lower is better).
  • YouTubeCovers: MOVE reached a MAP of 0.889, surpassing previous SOTA models like CQT-TPPNet (0.859).
  • Ablation Study: Removing the transposition-invariant architecture caused the biggest performance hit (MAP dropped from 0.575 to 0.154), proving that handling key changes is the most critical factor in VI.

Limitations and Future Work

While MOVE is highly effective, the authors note that some "fusion" methods (which combine multiple types of audio features) still offer unique advantages. A potential future direction is exploring how to incorporate multi-modal features (like melody and rhythm) into the embedding process without sacrificing the scalability of the Euclidean space. Additionally, while the system is scalable, the 16k-dimensional embeddings are relatively large; further research into dimensionality reduction or binary hashing could enhance its speed even further.

Summary updated

Abstract

The version identification (VI) task deals with the automatic detection of recordings that correspond to the same underlying musical piece. Despite many efforts, VI is still an open problem, with much room for improvement, specially with regard to combining accuracy and scalability. In this paper, we present MOVE, a musically-motivated method for accurate and scalable version identification. MOVE achieves state-of-the-art performance on two publicly-available benchmark sets by learning scalable embeddings in an Euclidean distance space, using a triplet loss and a hard triplet mining strategy. It improves over previous work by employing an alternative input representation, and introducing a novel technique for temporal content summarization, a standardized latent space, and a data augmentation strategy specifically designed for VI. In addition to the main results, we perform an ablation study to highlight the importance of our design choices, and study the relation between embedding dimensionality and model performance.
BibTeX
@inproceedings{yesiler2020accurate,
  author = {Furkan Yesiler and Joan Serrà and Emilia Gómez},
  title = {{Accurate and Scalable Version Identification Using Musically-Motivated Embeddings}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2020},
  url = {https://openreview.net/forum?id=X019EMymrf},
}

Loading related content...

Figure