Abstract
Music autotagging serves as a primary benchmark for general-purpose music AI, yet existing datasets often rely on noisy, crowdsourced labels. This paper transforms the MGPHot dataset—comprising expert musicological annotations for over 21,000 tracks—into a rigorous benchmark. By evaluating seven state-of-the-art audio representation models, the study reveals that top-performing models on generic datasets struggle with nuanced expert descriptors, identifying significant gaps in current music understanding capabilities.
Introduction: The Data Quality Problem in Autotagging
Music autotagging—the automatic assignment of labels such as genre, mood, or instrumentation—is the standard "proving ground" for music AI. However, the field is hampered by the quality of available data. Popular benchmarks like MagnaTagATune and MTG-Jamendo utilize crowdsourced or amateur tags, which are frequently inconsistent and limited to broad categories.
In contrast, the MGPHot dataset offers expert annotations for 21,320 Billboard Hot 100 tracks across 58 detailed attributes, including "Harmonic Sophistication" and "Vocal Grittiness." This research bridges the gap between these expert annotations and machine learning research by providing a curated audio retrieval pipeline and a standardized evaluation framework.
Methodology
To make MGPHot "ML-ready," the researchers developed a multi-stage pipeline focused on reproducibility and data integrity.
Audio Retrieval and Verification
Since the original MGPHot dataset does not include audio files due to legal restrictions, a pipeline was created to link tracks to YouTube URLs. This process utilized Regex matching and Large Language Models (specifically Qwen2.5-32B) to verify metadata against search results.
The pipeline ensured high-quality audio sources, with over 56% of final matches originating from official artist or label channels.
Dataset Partitioning and Probing
A sophisticated splitting procedure was designed to ensure balanced evaluation. The dataset was partitioned into train, validation, and test sets stratified by 58 expert features, release years, and audio "officialness." Crucially, the splits are artist-disjoint to prevent the model from memorizing specific artist styles.
To evaluate the models, a "frozen backbone" probing protocol was used. Seven pretrained encoders (e.g., Whisper, CLAP, MERT) had their weights frozen, while a simple two-layer MLP was trained on top to predict tags, measuring the inherent musical information captured by the pretrained representations.
Key Results and Analysis
The evaluation highlights a clear distinction between performance on generic datasets versus expert-annotated benchmarks.
Generic vs. Expert Performance
On the generic MagnaTagATune dataset, the MAEST model (trained on Discogs genre metadata) achieved a Mean Average Precision (MAP) of 0.493. However, on the expert-level MGPHot-tag dataset, CLAP (trained on audio-text pairs) took the lead with a MAP of 0.375, while MAEST’s performance dropped to 0.347. This suggests that models optimized for broad genre labels may fail to capture fine-grained musicological nuances.
Model-Specific Strengths
- Whisper: Originally designed for speech-to-text, Whisper performed exceptionally well in "Vocals" and "Lyrics" categories, indicating that speech-centric features are highly relevant for vocal-heavy music analysis.
- MERT: This self-supervised model performed best at continuous regression tasks (MGPHot-reg), achieving the lowest Root Mean Squared Error (0.164).
- Category Difficulty: Heatmap analysis revealed that "Instrument" and "Sonority" are relatively accessible for current models, whereas "Lyrics" and "Harmony" remain significant challenges.
Conclusion
The introduction of MGPHot as a benchmark reveals that no single model currently dominates all aspects of music understanding. The "difficulty gap" between generic and expert tags underscores the need for more nuanced datasets and evaluation protocols. Future work will explore fine-tuning these models and extending the benchmark to segment-level tagging and other MIR tasks like beat tracking.
Summary updated