Bytecover: Cover Song Identification Via Multi-Loss Training

Xingjian Du (杜行健), Xingjian Du, Zhesong Yu, Bilei Zhu, Xiaoou Chen, Zejun Ma

TL;DR

This paper introduces ByteCover, a novel deep learning system for cover song identification (CSI) that significantly improves accuracy by combining instance and batch normalization (IBN) with a multi-loss training strategy. ByteCover achieves state-of-the-art performance on multiple benchmark datasets, demonstrating its effectiveness in accurately identifying cover songs even with substantial musical variations.

What Problem Does This Paper Address?

Cover song identification (CSI) is the task of finding different versions of the same song within a large music database. This is a challenging problem because cover songs can vary significantly from their originals in terms of key, tempo, instrumentation, and even genre. Traditional methods, often relying on hand-crafted features and time-series matching, struggle to scale to the massive datasets common in modern music applications. While deep learning has shown promise, existing methods often treat CSI as either a classification or a metric learning problem, but not both simultaneously, potentially limiting their ability to capture both inter-song differences and intra-song similarities effectively.

Methods

ByteCover builds upon the ResNet50 architecture, incorporating two key innovations:

  • Instance-Batch Normalization (IBN) Blocks: The authors integrate Instance Normalization (IN) with Batch Normalization (BN) to create IBN blocks. This combination aims to learn features that are invariant to musical attribute changes (like key or tempo) while still preserving crucial version information. This is inspired by IBN's success in computer vision for learning style-invariant features.
  • Multi-Loss Training with BNNeck: ByteCover jointly optimizes a classification loss (Softmax Loss) and a metric learning loss (Triplet Loss) using a novel BNNeck module. This approach is designed to simultaneously enhance inter-class discrimination (distinguishing different songs) and intra-class compactness (grouping similar versions of the same song). The BNNeck normalizes features before the classification layer, making them more suitable for cosine similarity calculations during inference.

The input to the model is a Constant-Q Transform (CQT) spectrogram, which is processed through the ResNet-IBN backbone. A Generalized Mean Pooling (GeMPool) layer then compresses the feature maps into a fixed-length vector, which is used for both loss calculations and similarity comparisons.

Datasets

The paper evaluates ByteCover on four publicly available benchmark datasets:

  • SHS100K: A large dataset with 8858 songs and 108,523 recordings.
  • Youtube350: A dataset derived from YouTube, containing 50 compositions with multiple versions.
  • Covers80: A widely used benchmark with 80 songs, each having two covers.
  • Da-TACOS: A large dataset comprising 15,000 performances, with 13,000 belonging to 1,000 cliques.

Key Insights and Ideas

  • Synergy of Normalization: Combining IN and BN (IBN) proves beneficial for CSI, enabling the model to learn features robust to musical variations while retaining essential identification information. This suggests that normalization strategies can be tailored to specific domain challenges.
  • Power of Multi-Loss: Jointly training with classification and triplet losses is more effective than using either loss alone. This highlights the importance of addressing both discriminative and similarity aspects of the CSI problem simultaneously.
  • BNNeck for Feature Fusion: The BNNeck module effectively bridges the gap between classification and metric learning objectives, allowing for a more cohesive and powerful feature representation.
  • Scalability through Feature Representation: The fixed-length feature vectors generated by ByteCover are amenable to efficient indexing and retrieval, making the system scalable to large music databases.

Key Results

ByteCover achieved state-of-the-art performance across all evaluated datasets. Notably, on the Da-TACOS dataset, ByteCover achieved a mean average precision (mAP) of 71.4%, which is an 18.0% improvement over the previous best system. The paper also demonstrates that ByteCover can maintain competitive performance even with significantly smaller embedding vector sizes compared to other methods.

Limitations and Future Work

  • Computational Cost: While the paper focuses on efficiency, deep learning models, especially those with extensive normalization layers and multi-loss training, can still be computationally intensive.
  • Ablation Study Limitations: The ablation study in Table 3 shows aggregated improvements. While the paper attributes gains to IBN and BNNeck, a more granular breakdown of each component's contribution might offer deeper insights.
  • Future Work: The authors suggest that further exploration of different normalization techniques and loss function combinations could lead to even greater improvements. Additionally, investigating more sophisticated feature extraction and aggregation methods could be beneficial.

Summary updated

Abstract

We present in this paper ByteCover, which is a new feature learning method for cover song identification (CSI). ByteCover is built based on the classical ResNet model, and two major improvements are designed to further enhance the capability of the model for CSI. In the first improvement, we introduce the integration of instance normalization (IN) and batch normalization (BN) to build IBN blocks, which are major components of our ResNet-IBN model. With the help of the IBN blocks, our CSI model can learn features that are invariant to the changes of musical attributes such as key, tempo, timbre and genre, while preserving the version information. In the second improvement, we employ the BNNeck method to allow a multi-loss training and encourage our method to jointly optimize a classification loss and a triplet loss, and by this means, the inter-class discrimination and intra-class compactness of cover songs, can be ensured at the same time. A set of experiments demonstrated the effectiveness and efficiency of ByteCover on multiple datasets, and in the Da-TACOS dataset, ByteCover outperformed the best competitive system by 20.9\%.
BibTeX
@inproceedings{2021bytecover,
  author = {Xingjian Du (杜行健) and Xingjian Du and Zhesong Yu and Bilei Zhu and Xiaoou Chen and Zejun Ma},
  title = {{Bytecover: Cover Song Identification Via Multi-Loss Training}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2021},
  doi = {10.1109/icassp39728.2021.9414128},
}
Figure