Benchmarking Music Autotagging with MGPHot Expert Annotations vs. Generic Tag Datasets

Pedro Ramoneda, Pablo Alonso-Jiménez, Sergio Oramas, Xavier Serra, Dmitry BogdanovarXiv 2025 · DOI · arXiv

Abstract

Music autotagging serves as a primary benchmark for general-purpose music AI, yet existing datasets often rely on noisy, crowdsourced labels. This paper transforms the MGPHot dataset—comprising expert musicological annotations for over 21,000 tracks—into a rigorous benchmark. By evaluating seven state-of-the-art audio representation models, the study reveals that top-performing models on generic datasets struggle with nuanced expert descriptors, identifying significant gaps in current music understanding capabilities.

Introduction: The Data Quality Problem in Autotagging

Music autotagging—the automatic assignment of labels such as genre, mood, or instrumentation—is the standard "proving ground" for music AI. However, the field is hampered by the quality of available data. Popular benchmarks like MagnaTagATune and MTG-Jamendo utilize crowdsourced or amateur tags, which are frequently inconsistent and limited to broad categories.

In contrast, the MGPHot dataset offers expert annotations for 21,320 Billboard Hot 100 tracks across 58 detailed attributes, including "Harmonic Sophistication" and "Vocal Grittiness." This research bridges the gap between these expert annotations and machine learning research by providing a curated audio retrieval pipeline and a standardized evaluation framework.

Methodology

To make MGPHot "ML-ready," the researchers developed a multi-stage pipeline focused on reproducibility and data integrity.

Audio Retrieval and Verification

Since the original MGPHot dataset does not include audio files due to legal restrictions, a pipeline was created to link tracks to YouTube URLs. This process utilized Regex matching and Large Language Models (specifically Qwen2.5-32B) to verify metadata against search results.

The multi-stage audio retrieval pipeline for MGPHot.

The pipeline ensured high-quality audio sources, with over 56% of final matches originating from official artist or label channels.

Dataset Partitioning and Probing

A sophisticated splitting procedure was designed to ensure balanced evaluation. The dataset was partitioned into train, validation, and test sets stratified by 58 expert features, release years, and audio "officialness." Crucially, the splits are artist-disjoint to prevent the model from memorizing specific artist styles.

To evaluate the models, a "frozen backbone" probing protocol was used. Seven pretrained encoders (e.g., Whisper, CLAP, MERT) had their weights frozen, while a simple two-layer MLP was trained on top to predict tags, measuring the inherent musical information captured by the pretrained representations.

Key Results and Analysis

The evaluation highlights a clear distinction between performance on generic datasets versus expert-annotated benchmarks.

Generic vs. Expert Performance

On the generic MagnaTagATune dataset, the MAEST model (trained on Discogs genre metadata) achieved a Mean Average Precision (MAP) of 0.493. However, on the expert-level MGPHot-tag dataset, CLAP (trained on audio-text pairs) took the lead with a MAP of 0.375, while MAEST’s performance dropped to 0.347. This suggests that models optimized for broad genre labels may fail to capture fine-grained musicological nuances.

Mean Average Precision (MAP) scores for various models across MTG-Jamendo and MagnaTagATune datasets.

Model-Specific Strengths

  • Whisper: Originally designed for speech-to-text, Whisper performed exceptionally well in "Vocals" and "Lyrics" categories, indicating that speech-centric features are highly relevant for vocal-heavy music analysis.
  • MERT: This self-supervised model performed best at continuous regression tasks (MGPHot-reg), achieving the lowest Root Mean Squared Error (0.164).
  • Category Difficulty: Heatmap analysis revealed that "Instrument" and "Sonority" are relatively accessible for current models, whereas "Lyrics" and "Harmony" remain significant challenges.

Conclusion

The introduction of MGPHot as a benchmark reveals that no single model currently dominates all aspects of music understanding. The "difficulty gap" between generic and expert tags underscores the need for more nuanced datasets and evaluation protocols. Future work will explore fine-tuning these models and extending the benchmark to segment-level tagging and other MIR tasks like beat tracking.

Summary updated

Abstract

Music autotagging aims to automatically assign descriptive tags, such as genre, mood, or instrumentation, to audio recordings. Due to its challenges, diversity of semantic descriptions, and practical value in various applications, it has become a common downstream task for evaluating the performance of general-purpose music representations learned from audio data. We introduce a new benchmarking dataset based on the recently published MGPHot dataset, which includes expert musicological annotations, allowing for additional insights and comparisons with results obtained on common generic tag datasets. While MGPHot annotations have been shown to be useful for computational musicology, the original dataset neither includes audio nor provides evaluation setups for its use as a standardized autotagging benchmark. To address this, we provide a curated set of YouTube URLs with retrievable audio, and propose a train/val/test split for standardized evaluation, and precomputed representations for seven state-of-the-art models. Using these resources, we evaluated these models in MGPHot and standard reference tag datasets, highlighting key differences between expert and generic tag annotations. Altogether, our contributions provide a more advanced benchmarking framework for future research in music understanding.
BibTeX
@misc{ramoneda2025benchmarking,
  author = {Pedro Ramoneda and Pablo Alonso-Jiménez and Sergio Oramas and Xavier Serra and Dmitry Bogdanov},
  title = {{Benchmarking Music Autotagging with MGPHot Expert Annotations vs. Generic Tag Datasets}},
  year = {2025},
  eprint = {2509.06936},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2509.06936},
}
Figure