Audio-based Music Classification with a Pretrained Convolutional Network

Sander Dieleman, Philemon Brakel, Benjamin SchrauwenISMIR 2011 · OpenReview

Introduction

The emergence of the Million Song Dataset (MSD) in 2011 provided a massive corpus of audio features and metadata, presenting both an opportunity and a challenge for Music Information Retrieval (MIR). While supervised labels like genre or artist are available for only a fraction of the data, training deep learning models from scratch on such a scale is computationally intensive. This paper investigates whether a convolutional architecture, typically used for image processing, can be adapted to summarize musical features over hierarchical timescales (beats, bars, and themes). Furthermore, it explores the efficacy of unsupervised pretraining on the entire million-song corpus to enhance performance on specific downstream classification tasks.

Methodology

The authors propose a specialized Convolutional Multilayer Perceptron (MLP) architecture designed to capture the hierarchical nature of music.

Feature Engineering and Parallel Processing

Instead of raw audio, the model utilizes "beat-aligned" features derived from Echo Nest chroma (pitch) and timbre (texture) descriptors. These features are averaged within each beat to ensure rhythmic consistency. To prevent premature correlation between pitch and texture, the network employs two parallel paths for chroma and timbre features, allowing it to learn distinct representations for each before integration.

Hierarchical Architecture

The convolutional filters are structured to match musical timescales. The initial layers analyze 2-bar windows, which are subsequently aggregated into 8-bar "themes" through deeper layers of convolution and max-pooling.

Architecture of the Convolutional Multilayer Perceptron showing parallel processing of chroma and timbre features across musical timescales.

Unsupervised Pretraining

To leverage the unlabeled portion of the MSD, the network is first trained as a Convolutional Deep Belief Network (CDBN) using Contrastive Divergence. This phase allows the model to learn a general "language" of musical features. These pretrained weights then initialize the MLP for supervised finetuning on labeled subsets.

Experimental Setup

The model was evaluated on three subsets of the MSD:

  • Artist Recognition: 5,000 tracks from the 50 most frequent artists.
  • Genre Recognition: 5,000 tracks across 20 common genres.
  • Key Detection: 3,000 tracks with high-confidence key annotations.

Results and Discussion

The experimental results demonstrate that the convolutional architecture effectively summarizes temporal audio data.

Comparison of classification accuracy across different tasks, models, and training durations.

Performance and Convergence

The Convolutional MLP significantly outperformed baseline models in high-level tasks. In genre recognition, it achieved 29.52% accuracy compared to 25.90% for windowed logistic regression. Pretraining proved particularly beneficial for artist recognition, where labeled data is often scarce, yielding 35.74% accuracy. Notably, pretrained models exhibited faster convergence, reaching peak performance in fewer epochs than randomly initialized counterparts.

Task Complexity

The study highlights a distinction between "low-level" and "high-level" tasks. Key Detection, being closely tied to raw chroma features, was handled well by simpler models (86.53% accuracy). Conversely, Genre and Artist Recognition required the hierarchical abstractions provided by the deep convolutional layers to achieve competitive results.

Conclusion

This work demonstrates that unsupervised pretraining on large-scale datasets like the MSD provides a significant "head start" for music classification tasks, improving both accuracy and training efficiency. Future research may address label noise in the MSD and explore alternative visible units in the RBM to better model continuous audio features.

Reference: Dieleman, S., Brakel, P., & Schrauwen, B. (2011). Audio-based music classification with a pretrained convolutional network. In Proceedings of the 12th International Society for Music Information Retrieval Conference (ISMIR 2011).

Summary updated

BibTeX
@inproceedings{dieleman2011audio,
  author = {Sander Dieleman and Philemon Brakel and Benjamin Schrauwen},
  title = {{Audio-based Music Classification with a Pretrained Convolutional Network}},
  booktitle = {ISMIR},
  year = {2011},
  url = {https://openreview.net/forum?id=dqLLV3HKf2},
}
Figure