mir_ref: A Representation Evaluation Framework for Music Information Retrieval Tasks

Christos Plachouras, Pablo Alonso-Jiménez, Dmitry BogdanovarXiv 2023 · DOI · arXiv

Introduction

In the field of Music Information Retrieval (MIR), the paradigm is shifting from hand-crafted features toward representation learning. Deep learning models are now used to generate compact "embeddings" that encapsulate musical characteristics. However, evaluating these embeddings remains a fragmented process. Researchers often use inconsistent preprocessing, varied downstream classifiers, or private datasets, making it difficult to compare models fairly. Existing benchmarks like HEAR or MARBLE, while useful, often function as "black boxes" with fixed pipelines that limit deep experimentation.

To address these challenges, we introduce mir_ref, an open-source Python framework designed to standardize and bring transparency to music audio representation evaluation. It provides a "local-first" orchestration tool that allows researchers to conduct reproducible experiments across diverse MIR tasks and real-world audio conditions.

The mir_ref Framework

mir_ref is a configuration-based framework that manages the entire evaluation pipeline, from data loading to metric calculation. Its modular design integrates industry-standard tools to ensure consistency and ease of use.

The mir_ref evaluation pipeline, illustrating the flow from dataset preprocessing and deformation to feature extraction, downstream training, and final evaluation.

Key features of the framework include:

  • Local-First Orchestration: Runs on local hardware, enabling rapid iteration during model development.
  • Modular Components: Leverages mirdata for standardized dataset loading, audiomentations for applying audio deformations (e.g., noise, pitch shifting), and mir_eval for metric consistency.
  • Transparent Downstream Models: Allows users to swap between simple linear classifiers and complex Multi-Layer Perceptrons (MLPs) to determine if information is linearly accessible or requires non-linear decoding.
  • Reproducibility: Experiments are defined via configuration files, ensuring that testing conditions can be shared and replicated exactly.

Experimental Setup and Datasets

The framework supports a wide array of MIR benchmarks out of the box. To demonstrate its utility, we evaluated seven popular representation models across six diverse datasets covering genre, instrumentation, mood, and vocal techniques.

A comprehensive overview of the models and datasets supported and evaluated within the mir_ref framework.

Supported datasets include MTG-Jamendo (tagging), GTZAN and FMA (genre), VocalSet (singer identity/technique), NSynth (pitch/instrument), and Beatport EDM (key detection).

Key Insights and Results

Our initial evaluations using mir_ref revealed several critical insights into the current state of music representations:

  • The Robustness Gap: Most representations are fragile under real-world conditions. While they handle MP3 compression well, performance drops significantly when faced with white noise or gain reduction. For example, in instrument recognition tasks, models like CLMR and OpenL3 faltered at 0 dB SNR.
  • Linear vs. Non-Linear Information: The choice of downstream model is crucial. For singer identification (VocalSet), using an MLP instead of a linear classifier doubled the F1 score for models like NeuralFP. This indicates that while the "identity" information is present in the embedding, it is not always linearly separable.
  • The Pitch Problem: General-purpose embeddings often struggle with pitch. In pitch class classification on the TinySOL dataset, most models failed, with only MERT and NeuralFP showing significant ability to discern pitch.
  • Model Versatility: MERT emerged as one of the most versatile models, particularly excelling in tasks requiring fine-grained pitch information.

Conclusion

mir_ref provides the MIR community with a much-needed tool for transparent and standardized evaluation. By moving beyond static leaderboards, it enables researchers to understand why models succeed or fail. Future work will focus on enhancing explainability through interactive analysis tools and expanding the framework's modular library with community-contributed datasets and tasks.

Summary updated

Abstract

Music Information Retrieval (MIR) research is increasingly leveraging representation learning to obtain more compact, powerful music audio representations for various downstream MIR tasks. However, current representation evaluation methods are fragmented due to discrepancies in audio and label preprocessing, downstream model and metric implementations, data availability, and computational resources, often leading to inconsistent and limited results. In this work, we introduce mir_ref, an MIR Representation Evaluation Framework focused on seamless, transparent, local-first experiment orchestration to support representation development. It features implementations of a variety of components such as MIR datasets, tasks, embedding models, and tools for result analysis and visualization, while facilitating the implementation of custom components. To demonstrate its utility, we use it to conduct an extensive evaluation of several embedding models across various tasks and datasets, including evaluating their robustness to various audio perturbations and the ease of extracting relevant information from them.
BibTeX
@misc{plachouras2023mir,
  author = {Christos Plachouras and Pablo Alonso-Jiménez and Dmitry Bogdanov},
  title = {{mir\_ref: A Representation Evaluation Framework for Music Information Retrieval Tasks}},
  year = {2023},
  eprint = {2312.05994},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2312.05994},
}
Figure