Introduction
In the field of Music Information Retrieval (MIR), the paradigm is shifting from hand-crafted features toward representation learning. Deep learning models are now used to generate compact "embeddings" that encapsulate musical characteristics. However, evaluating these embeddings remains a fragmented process. Researchers often use inconsistent preprocessing, varied downstream classifiers, or private datasets, making it difficult to compare models fairly. Existing benchmarks like HEAR or MARBLE, while useful, often function as "black boxes" with fixed pipelines that limit deep experimentation.
To address these challenges, we introduce mir_ref, an open-source Python framework designed to standardize and bring transparency to music audio representation evaluation. It provides a "local-first" orchestration tool that allows researchers to conduct reproducible experiments across diverse MIR tasks and real-world audio conditions.
The mir_ref Framework
mir_ref is a configuration-based framework that manages the entire evaluation pipeline, from data loading to metric calculation. Its modular design integrates industry-standard tools to ensure consistency and ease of use.
Key features of the framework include:
- Local-First Orchestration: Runs on local hardware, enabling rapid iteration during model development.
- Modular Components: Leverages
mirdatafor standardized dataset loading,audiomentationsfor applying audio deformations (e.g., noise, pitch shifting), andmir_evalfor metric consistency. - Transparent Downstream Models: Allows users to swap between simple linear classifiers and complex Multi-Layer Perceptrons (MLPs) to determine if information is linearly accessible or requires non-linear decoding.
- Reproducibility: Experiments are defined via configuration files, ensuring that testing conditions can be shared and replicated exactly.
Experimental Setup and Datasets
The framework supports a wide array of MIR benchmarks out of the box. To demonstrate its utility, we evaluated seven popular representation models across six diverse datasets covering genre, instrumentation, mood, and vocal techniques.
Supported datasets include MTG-Jamendo (tagging), GTZAN and FMA (genre), VocalSet (singer identity/technique), NSynth (pitch/instrument), and Beatport EDM (key detection).
Key Insights and Results
Our initial evaluations using mir_ref revealed several critical insights into the current state of music representations:
- The Robustness Gap: Most representations are fragile under real-world conditions. While they handle MP3 compression well, performance drops significantly when faced with white noise or gain reduction. For example, in instrument recognition tasks, models like CLMR and OpenL3 faltered at 0 dB SNR.
- Linear vs. Non-Linear Information: The choice of downstream model is crucial. For singer identification (VocalSet), using an MLP instead of a linear classifier doubled the F1 score for models like NeuralFP. This indicates that while the "identity" information is present in the embedding, it is not always linearly separable.
- The Pitch Problem: General-purpose embeddings often struggle with pitch. In pitch class classification on the TinySOL dataset, most models failed, with only MERT and NeuralFP showing significant ability to discern pitch.
- Model Versatility: MERT emerged as one of the most versatile models, particularly excelling in tasks requiring fine-grained pitch information.
Conclusion
mir_ref provides the MIR community with a much-needed tool for transparent and standardized evaluation. By moving beyond static leaderboards, it enables researchers to understand why models succeed or fail. Future work will focus on enhancing explainability through interactive analysis tools and expanding the framework's modular library with community-contributed datasets and tasks.
Summary updated