Learning to Recognize Musical Genre from Audio

Michaël Defferrard, Sharada P. Mohanty, Sean F. Carroll, Marcel Salathé

Abstract

This paper summarizes the "Learning to Recognize Musical Genre from Audio" challenge, an initiative designed to advance Music Information Retrieval (MIR) using open-source data. By leveraging the Free Music Archive (FMA) dataset and a rigorous two-round submission process involving Dockerized models, the organizers established a new benchmark for genre classification. The challenge promoted reproducible research in a field often hampered by copyright restrictions, resulting in high-performing models that utilize ensemble methods and multi-task learning.

Introduction

As digital music libraries have expanded to include millions of tracks, automated organization and recommendation systems have become critical. Musical Genre Recognition (MGR) is a foundational task for these systems. However, MIR research faces a significant hurdle: copyright. Most commercial music cannot be legally shared for research, leading to "closed" datasets that prevent verification or collaborative building. Furthermore, genre is a subjective concept with fluid boundaries between categories like rock, pop, and electronic music. This challenge addressed these issues by hosting a competition based entirely on Creative Commons-licensed audio from the Free Music Archive.

Methodology

Experimental Design

The challenge was structured in two distinct rounds to ensure the integrity and reproducibility of the results:

  1. Round 1 (The Leaderboard): Participants were provided with a test set of 30,000 clips. They uploaded predictions to the crowdAI platform, receiving immediate feedback and ranking based on their performance.
  2. Round 2 (The "Truth" Round): To prevent overfitting or manual labeling, the top participants submitted their actual code and trained models as Git repositories. The organizers built Docker containers from these repositories and evaluated them against a completely new, unseen test set sampled from recent FMA additions.

Evaluation Metrics

The primary evaluation metric was mean log loss, which penalizes confident but incorrect predictions. The secondary metric was the mean F1 score, providing a balance between precision and recall.

Datasets

The challenge utilized the Free Music Archive (FMA) dataset, a 917 GiB open-source repository of high-quality audio. The training set consisted of the "FMA medium" subset (25,000 30-second clips across 16 unbalanced genres). Round 2 used a brand-new set of tracks to ensure zero data leakage.

The progression of top F1-scores over the duration of the competition shows steady improvement across various participant models.

Results and Discussion

Participation and Performance

The challenge engaged 246 participants, resulting in 576 successfully graded submissions. Over 60% of these submissions outperformed the baseline model provided by the organizers. The best-performing solution achieved a mean F1 score of 0.909 and a mean log loss of 0.330.

A joint plot showing the distribution of Log Loss versus F1-Score for all submissions, illustrating the performance landscape and the correlation between the two primary metrics.

Key Insights

  • Reproducibility via Docker: Requiring code submissions ensured that winning models were functional and reproducible, rather than just static prediction files.
  • Ensembles vs. Deep Learning: One notable finding was that ensemble methods, specifically XGBoost using high-level features extracted via the Essentia library, outperformed standard Convolutional Neural Networks (CNNs) trained on spectrograms.
  • Artist Identity as a Proxy: Successful approaches utilized multi-task learning to predict both artist identity and genre. Because artist identity is objective, features learned for artist recognition significantly improved the accuracy of subjective genre classification through transfer learning.

Conclusion

While genre classification remains difficult due to inherent categorical ambiguity and unbalanced datasets, this challenge provided a robust, legal, and open benchmark. By releasing the code from second-round participants as open-source, the organizers have provided the MIR community with a suite of high-performing baselines, moving the field toward more transparent and collaborative "Open Science."

Summary updated

BibTeX
@inproceedings{defferrard2018learning,
  author = {Michaël Defferrard and Sharada P. Mohanty and Sean F. Carroll and Marcel Salathé},
  title = {{Learning to Recognize Musical Genre from Audio}},
  booktitle = {The Web Conference},
  year = {2018},
  doi = {10.1145/3184558.3192310},
}
Figure