Introduction
Deep learning models for music classification typically require vast amounts of labeled data, which is expensive and time-consuming to acquire. Furthermore, models trained on specific datasets often struggle to generalize. This paper investigates Self-Supervised Learning (SSL) as a solution, where a model learns representations from unlabeled data by solving "pretext tasks" that capture the inherent structure of music, such as timbre, harmony, and rhythm.
Methodology
The researchers propose a two-stage framework involving pre-training and downstream evaluation.
Pre-training Pipeline
The core of the approach is an encoder trained on the "Music" subset of AudioSet (approx. 83 hours). The encoder processes raw audio waveforms and is tasked with reconstructing several hand-crafted musical features through multiple "worker" networks. These features include:
- MFCCs: Capturing timbre.
- Chroma: Capturing harmonic content.
- Tempogram: Capturing rhythmic patterns.
- LPS & Prosody: Standard audio baselines.
Architectural Innovations
The study utilizes the PASE+ architecture, which improves upon the original PASE by incorporating SincNet layers for raw waveform processing and Quasi-Recurrent Neural Networks (QRNN) to model long-term temporal dependencies. Additionally, a loss re-weighting mechanism was implemented to prevent simpler tasks (like power spectrum reconstruction) from dominating the learning process, ensuring the model also learns complex features like rhythm.
Experimental Setup
The pre-trained encoder was evaluated on three downstream classification tasks:
- OpenMIC: Instrument recognition (15k samples).
- Extended Ballroom: Dance genre classification (e.g., Waltz, Tango).
- FMA Small: General music genre classification.
Key Results
The experimental results highlight the effectiveness of music-specific pretext tasks and the PASE+ architecture.
- Architecture Performance: PASE+ consistently outperformed the standard PASE model across all datasets, validating the use of recurrent layers and skip connections for music audio.
- Impact of Rhythmic Features: The inclusion of the Tempogram worker led to a 14% improvement in dance genre classification on the Extended Ballroom dataset, underscoring the necessity of rhythmic supervision for genre-related tasks.
- Task Complementarity: Combining multiple pretext tasks yielded the best generalization. While MFCCs aided instrument identification, Chroma features were essential for distinguishing genres with similar timbres but different harmonic structures.
- Fine-tuning: Allowing the encoder weights to update during downstream tasks (fine-tuning) generally outperformed a "frozen" encoder, particularly when labeled data exceeded 1,000 samples.
Conclusion
This work demonstrates that multi-task self-supervised pre-training significantly enhances music classification, especially when labeled data is scarce. By balancing various musical pretext tasks, the model develops a robust understanding of timbre, harmony, and rhythm. Future research may explore data-driven pretext tasks, such as contrastive learning, to further reduce reliance on hand-crafted features.
Summary updated