Temporal Pyramid Pooling Convolutional Neural Network for Cover Song Identification

Zhesong Yu, Xiaoshuo Xu, Xiaoou Chen, Deshun YangIJCAI 2019 · DOI

TL;DR

This paper introduces a novel Convolutional Neural Network (CNN) architecture, CQT-TPPNet, enhanced with Temporal Pyramid Pooling (TPP) and a multi-length training strategy, to tackle the challenging problem of cover song identification. The approach learns key-invariant representations of music, outperforming existing methods on several datasets with significantly lower computational cost.

What Problem Does This Paper Address?

Cover song identification (CSI) aims to find different versions of the same song within a large music database. This is crucial for applications like music licensing, clustering, and recommendation. However, CSI is notoriously difficult due to inherent variations in cover songs, such as changes in key (transposition), tempo, and structural alterations (e.g., skipped intros, repeated choruses). Traditional methods often rely on hand-crafted features and complex sequence alignment algorithms (like Dynamic Time Warping), which are computationally expensive and struggle with scalability to large datasets. While some recent approaches use machine learning, they often sacrifice temporal information or require extensive manual feature engineering. This paper seeks to bridge this gap by developing an efficient and robust deep learning model for CSI that can handle these musical variations.

Methods

The core of the proposed method, CQT-TPPNet, is a CNN designed to learn key-invariant music representations. The input to the network is the Constant-Q Transform (CQT) of the audio, which is a low-level descriptor that preserves more harmonic information than commonly used Chroma features.

Key methodological contributions include:

  • Key-Invariant Representation Learning: The CNN is trained to learn features that are robust to key transpositions, analogous to how CNNs learn shift-invariant features in image recognition. This is achieved by treating different versions of the same song as belonging to the same class during training.
  • Temporal Pyramid Pooling (TPP): To address the variable length of music pieces and extract information at multiple temporal scales, TPP is employed. This technique, inspired by its success in computer vision, partitions the feature maps from the CNN into different levels and applies max-pooling. The pooled features from these levels are then concatenated to form a fixed-dimensional representation, regardless of the input song's duration. This avoids the need for computationally expensive time-stretching or padding.
  • Multi-length Training Strategy: To further enhance robustness against duration variations and improve generalization, a multi-length training scheme is introduced. During training, audio segments are randomly cropped to different lengths (e.g., 100s, 150s, 200s), effectively augmenting the dataset and preventing the model from becoming biased towards a specific input length. This also simulates performance variations like skipped sections.

Datasets

The paper utilizes the following datasets for training and evaluation:

  • Second Hand Songs 100K (SHS100K): A large dataset containing 8858 songs and over 108,000 recordings. It was split into training, validation, and testing sets (8:1:1 ratio). For training, a subset of songs with more than five covers was used to create a more balanced training set. The test set was used to compute a large similarity matrix for evaluation.
  • YouTube: A dataset collected by Silva et al. (2016), consisting of 50 compositions with 7 versions each (350 recordings total). The original training/testing split was used for comparison.
  • Covers80: A benchmark dataset with 80 songs, each having 2 covers. This dataset was used for evaluating similarity across all pairs.

Key Insights and Ideas

  • CNNs are naturally suited for key-invariant music representation: The paper demonstrates that CNNs, when applied to music spectrograms like CQT, can learn representations that are inherently robust to key transpositions, a significant challenge in CSI.
  • TPP is crucial for handling variable-length music data: By aggregating information across different temporal scales, TPP allows the network to produce fixed-size representations from variable-length inputs, a critical step for efficient retrieval without resorting to signal manipulation.
  • Multi-length training significantly boosts robustness: The proposed training strategy not only increases the effective dataset size but also makes the model more resilient to variations in song duration and structural changes, mimicking real-world performance variations.
  • Low-level features (CQT) combined with deep learning outperform hand-crafted methods: The success of CQT-TPPNet suggests that deep learning can effectively learn rich representations from raw audio features, surpassing traditional feature engineering approaches.

Key Results

  • State-of-the-art performance: CQT-TPPNet achieved the highest precision (MAP) on the YouTube and Covers80 datasets, outperforming several existing methods. For example, on Covers80, it achieved a MAP of 0.744.
  • High efficiency: The model demonstrates significantly lower query time compared to sequence alignment-based methods. On Covers80, it processed queries in approximately 0.06 ms. The authors estimate it can handle million-song datasets within seconds.
  • Robustness: The model showed good performance even with simulated musical variations like key transposition (up to ±12 semitones) and tempo changes (within ±20%), with only a slight degradation in accuracy.
  • Effectiveness of TPP: Experiments showed that TPP consistently improved performance over CQT-Net variants that used fixed-length inputs without TPP. Using multiple pyramid levels in TPP yielded better results than a single level.

Limitations and Future Work

  • Performance degradation on larger, imbalanced datasets: While CQT-TPPNet performs well, accuracy decreases on the large SHS100K-TEST dataset, suggesting that highly imbalanced datasets with many similar-sounding songs pose a greater challenge.
  • Difficulty with certain music styles: The model exhibits higher error rates for genres like rock music and songs with strong accompaniments, likely due to the presence of non-harmonic content and complex instrumentation that can obscure the core musical structure.
  • Future Work: The authors suggest exploring methods to extract representations that are more robust to these challenging musical characteristics. Additionally, they note that the model's ability to retrieve music with similar styles, not just covers, suggests potential for content-based music recommendation.

Summary updated

BibTeX
@inproceedings{yu2019temporal,
  author = {Zhesong Yu and Xiaoshuo Xu and Xiaoou Chen and Deshun Yang},
  title = {{Temporal Pyramid Pooling Convolutional Neural Network for Cover Song Identification}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2019},
  doi = {10.24963/ijcai.2019/673},
}
Figure