An Experimental Comparison Of Multi-view Self-supervised Methods For Music Tagging

Gabriel Meseguer-Brocal, Dorian Desblancs, Romain Hennequin

Abstract

Self-supervised learning (SSL) has emerged as a powerful paradigm for training deep learning models without the need for extensive human-labeled datasets. While non-contrastive SSL methods have gained traction in computer vision, their effectiveness in the audio domain, specifically for music tagging, remains less understood. This study evaluates five prominent multi-view SSL methods—Contrastive Learning, BYOL, Clustering, Barlow Twins, and VICReg—using a large-scale pre-training dataset of 4 million tracks. Our findings demonstrate that contrastive learning remains the superior and most stable method for music audio, consistently outperforming alternatives across multiple benchmarks and showing exceptional robustness in low-data regimes.

Introduction

Deep learning models for music tagging—identifying genres, moods, or instruments—typically require massive amounts of labeled data, which is expensive and subjective to produce. Self-supervised learning (SSL) offers a solution by leveraging unlabeled data through pretext tasks. While contrastive learning is the current standard in audio, recent developments in computer vision have introduced non-contrastive alternatives. This paper investigates whether these newer methods translate effectively to music tagging or if contrastive learning remains the optimal choice.

Methodology

To ensure a fair comparison, the researchers utilized a consistent "backbone" architecture: a lightweight ResNet with 2.8 million parameters. The study focused on multi-view SSL, where the model processes two different four-second snippets (an "anchor" and a "positive") from the same track. Five distinct pretext tasks were evaluated:

  1. Contrastive Learning: Maximizes similarity between views of the same song while minimizing similarity between different songs.
  2. BYOL (Bootstrap Your Own Latent): A teacher-student framework where the model predicts its own representation of a different view.
  3. Clustering: Assigns embeddings to "prototypes" or clusters to learn categorical structures.
  4. Barlow Twins: Reduces redundancy by making the cross-correlation matrix of embeddings approach the identity matrix.
  5. VICReg: Balances Variance, Invariance, and Covariance to prevent representational collapse.

Experimental Setup

The models were pre-trained on an in-house Deezer dataset comprising approximately 4 million tracks from diverse playlists. Evaluation was conducted on five standard downstream benchmarks: MSD100 (Million Song Dataset subset), MTAT (MagnaTagATune), and three Jamendo subsets (JamTop50, JamInstrument, and JamMood).

Results and Analysis

Performance and Stability

Contrastive learning emerged as the clear winner, achieving the highest mean Average Precision (mAP) and ROC scores across all datasets. Unlike computer vision, where non-contrastive methods often match contrastive performance, music audio seems to benefit significantly from the explicit negative sampling inherent in contrastive tasks. Clustering generally ranked second, while BYOL performed the poorest, likely due to the restrictive nature of its predictive task for complex audio signals.

Comparison of mAP and ROC scores across five benchmark datasets for different SSL methods.

Data Efficiency

A critical finding was the efficiency of these SSL models in low-resource scenarios. The models achieved respectable performance using only 1% of the downstream training data and reached near-optimal results with just 10%. This suggests that SSL-pre-trained models are highly effective for niche or under-labeled musical styles.

Performance scaling of SSL methods as a function of available downstream training data percentage.

Observed Failure Modes

The study identified specific challenges for non-contrastive methods in audio. VICReg and BYOL were highly sensitive to hyperparameter tuning, with default image-based settings often failing. Furthermore, a unique "uncharted collapse" was observed in clustering, where the model utilized only a small fraction of available clusters, limiting its representational capacity.

Conclusion

Contrastive learning remains the "gold standard" for self-supervised music tagging. It offers superior performance, greater stability, and requires less hyperparameter optimization than newer non-contrastive methods. While non-contrastive approaches might improve with audio-specific data augmentations, current results favor contrastive frameworks for robust music representation learning. Future work will address the Western bias in current datasets and explore hybrid pretext tasks to further enhance model generalization.

Reference: Gabriel Meseguer-Brocal, Dorian Desblancs, and Romain Hennequin. "An Experimental Comparison Of Multi-view Self-supervised Methods For Music Tagging." Deezer Research, 2024.

Summary updated

BibTeX
@inproceedings{meseguerbrocal2024experimental,
  author = {Gabriel Meseguer-Brocal and Dorian Desblancs and Romain Hennequin},
  title = {{An Experimental Comparison Of Multi-view Self-supervised Methods For Music Tagging}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2024},
  doi = {10.1109/icassp48485.2024.10447473},
}
Figure