Multi-Task Self-Supervised Pre-Training for Music Classification

Ho-Hsiang Wu, Chieh-Chi Kao, Qingming Tang, Ming Sun, Brian McFee, Juan Pablo Bello, Chao Wang

Introduction

Deep learning models for music classification typically require vast amounts of labeled data, which is expensive and time-consuming to acquire. Furthermore, models trained on specific datasets often struggle to generalize. This paper investigates Self-Supervised Learning (SSL) as a solution, where a model learns representations from unlabeled data by solving "pretext tasks" that capture the inherent structure of music, such as timbre, harmony, and rhythm.

Methodology

The researchers propose a two-stage framework involving pre-training and downstream evaluation.

Pre-training Pipeline

The core of the approach is an encoder trained on the "Music" subset of AudioSet (approx. 83 hours). The encoder processes raw audio waveforms and is tasked with reconstructing several hand-crafted musical features through multiple "worker" networks. These features include:

  • MFCCs: Capturing timbre.
  • Chroma: Capturing harmonic content.
  • Tempogram: Capturing rhythmic patterns.
  • LPS & Prosody: Standard audio baselines.
The multi-task self-supervised pre-training pipeline.

Architectural Innovations

The study utilizes the PASE+ architecture, which improves upon the original PASE by incorporating SincNet layers for raw waveform processing and Quasi-Recurrent Neural Networks (QRNN) to model long-term temporal dependencies. Additionally, a loss re-weighting mechanism was implemented to prevent simpler tasks (like power spectrum reconstruction) from dominating the learning process, ensuring the model also learns complex features like rhythm.

Experimental Setup

The pre-trained encoder was evaluated on three downstream classification tasks:

  1. OpenMIC: Instrument recognition (15k samples).
  2. Extended Ballroom: Dance genre classification (e.g., Waltz, Tango).
  3. FMA Small: General music genre classification.

Key Results

The experimental results highlight the effectiveness of music-specific pretext tasks and the PASE+ architecture.

Macro F1 performance of PASE vs. PASE+ on downstream tasks.
  • Architecture Performance: PASE+ consistently outperformed the standard PASE model across all datasets, validating the use of recurrent layers and skip connections for music audio.
  • Impact of Rhythmic Features: The inclusion of the Tempogram worker led to a 14% improvement in dance genre classification on the Extended Ballroom dataset, underscoring the necessity of rhythmic supervision for genre-related tasks.
  • Task Complementarity: Combining multiple pretext tasks yielded the best generalization. While MFCCs aided instrument identification, Chroma features were essential for distinguishing genres with similar timbres but different harmonic structures.
  • Fine-tuning: Allowing the encoder weights to update during downstream tasks (fine-tuning) generally outperformed a "frozen" encoder, particularly when labeled data exceeded 1,000 samples.

Conclusion

This work demonstrates that multi-task self-supervised pre-training significantly enhances music classification, especially when labeled data is scarce. By balancing various musical pretext tasks, the model develops a robust understanding of timbre, harmony, and rhythm. Future research may explore data-driven pretext tasks, such as contrastive learning, to further reduce reliance on hand-crafted features.

Summary updated

Abstract

Deep learning is very data hungry, and supervised learning especially requires massive labeled data to work well. Machine listening research often suffers from limited labeled data problem, as human annotations are costly to acquire, and annotations for audio are time consuming and less intuitive. Besides, models learned from labeled dataset often embed biases specific to that particular dataset. Therefore, unsupervised learning techniques become popular approaches in solving machine listening problems. Particularly, a self-supervised learning technique utilizing reconstructions of multiple hand-crafted audio features has shown promising results when it is applied to speech domain such as emotion recognition and automatic speech recognition (ASR). In this paper, we apply self-supervised and multi-task learning methods for pre-training music encoders, and explore various design choices including encoder architectures, weighting mechanisms to combine losses from multiple tasks, and worker selections of pretext tasks. We investigate how these design choices interact with various downstream music classification tasks. We find that using various music specific workers altogether with weighting mechanisms to balance the losses during pre-training helps improve and generalize to the downstream tasks.
BibTeX
@inproceedings{wu2021multi,
  author = {Ho-Hsiang Wu and Chieh-Chi Kao and Qingming Tang and Ming Sun and Brian McFee and Juan Pablo Bello and Chao Wang},
  title = {{Multi-Task Self-Supervised Pre-Training for Music Classification}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2021},
  url = {https://openreview.net/forum?id=raGZU6_zaW5},
}
Figure