MID-FiLD: MIDI Dataset for Fine-Level Dynamics

Jesung Ryu, Seungyeon Rhyu, Hong-Gyu Yoon, Eunchong Kim, Ju Young Yang, Taehyun KimAAAI 2024 · DOI

TL;DR

This paper introduces MID-FiLD, a novel MIDI dataset meticulously crafted by professional composers, featuring fine-grained expressive dynamics and rich metadata. This dataset addresses the critical lack of expressive nuance in current music generation models, enabling more human-like and controllable musical outputs.

What Problem Does This Paper Address?

Generating music that truly captures the expressiveness of human performance remains a significant challenge. While models can produce correct notes and rhythms, they often struggle with subtle variations in dynamics (loudness), phrasing, and timbre that make music sound alive. Existing datasets often focus on note-level velocity, which is insufficient for instruments that can vary loudness within a single note (e.g., a crescendo on a violin). Furthermore, these datasets often lack crucial contextual information like the intended mood or the role of an instrument within a piece, which human composers naturally consider when shaping dynamics. This gap limits the ability of AI to create truly compelling and nuanced musical expressions.

Methods

The authors propose a token-based input encoding strategy for their generative model, inspired by Natural Language Processing techniques. This approach represents note sequences, metadata, and fine-level dynamics (CC#1 values) as sequences of integers. Specifically, they introduce a "position end" token to better capture note durations and use CC#1 (modulation wheel) values to represent continuous changes in loudness. Their generative model is a standard transformer architecture, trained to predict CC#1 values conditioned on the note sequence and metadata. For evaluation, they introduce metrics for "fidelity" (RMSE of predicted CC#1 values against ground truth) and "controllability" (differences in min/max CC#1 values and mood classification accuracy).

Datasets

The paper introduces MID-FiLD, a new dataset comprising 4,422 MIDI samples. These samples were created by professional composers and include:

  • MIDI Notes: The core musical information.
  • Fine-level Expressive Dynamics: Represented by CC#1 (modulation wheel) values, capturing continuous loudness variations.
  • Metadata: Including instrument type (18 categories), track role (e.g., melody, bass, accompaniment), and mood (19 categories).

This dataset is compared against existing MIDI datasets like MAESTRO, MusicNet, URMP, Lakh MIDI, and ComMU, highlighting MID-FiLD's unique contribution in providing both fine-level dynamics and comprehensive metadata.

Key Insights and Ideas

  • Fine-level dynamics are crucial: Simply relying on note-level velocity is insufficient for capturing the expressive nuances of many instruments. CC#1 values offer a more granular control over loudness.
  • Metadata significantly impacts dynamics: The study reveals strong correlations between metadata (mood, instrument, track role) and expressive dynamics. This suggests that incorporating such contextual information is vital for generating realistic and controllable music.
  • Mood is a strong predictor of dynamics: The dataset analysis shows that mood, particularly high arousal and low valence, is associated with increased dynamics, aligning with psychoacoustic research.
  • Controllability is enhanced by metadata: Models trained with more metadata exhibit better controllability, meaning they can generate dynamics closer to the intended range and emotional context.

Key Results

  • Fidelity: The model trained with all metadata achieved the lowest RMSE (21.6794), indicating the highest fidelity in predicting CC#1 values. Removing any metadata item led to a statistically significant increase in RMSE.
  • Controllability (Min/Max Difference): The model with all metadata showed the smallest differences in min/max CC#1 values (10.9861 and 17.2546), demonstrating superior control over the dynamic range.
  • Mood Classification: The ground truth achieved the highest accuracy (0.7245, 0.8495, 0.8056 for 4Q, V, A respectively). The model with all metadata performed best among the trained models, particularly in distinguishing valence (V). Notably, the model trained without mood metadata performed worse than even the model without any metadata for arousal (A), underscoring mood's importance.
  • Ablation Study: Removing metadata items one by one consistently degraded both fidelity and controllability, emphasizing the value of each metadata component. Track role and mood were found to be particularly influential.

Limitations and Future Work

  • Limitations: The paper acknowledges that the current generative model is a baseline and that further exploration of representations and architectures is possible. The "min&max CC#1 value" metadata, while useful, is derived from the CC#1 values themselves, which could be seen as a form of data leakage if not handled carefully in downstream tasks.
  • Future Work: The authors suggest that their dataset and methodologies can benefit a wide range of models and contribute to the field of music generation through innovative representations. They envision further research exploring more sophisticated models and representations that can leverage the rich metadata and fine-level dynamics provided by MID-FiLD.

Summary updated

BibTeX
@inproceedings{ryu2024mid,
  author = {Jesung Ryu and Seungyeon Rhyu and Hong-Gyu Yoon and Eunchong Kim and Ju Young Yang and Taehyun Kim},
  title = {{MID-FiLD: MIDI Dataset for Fine-Level Dynamics}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2024},
  doi = {10.1609/aaai.v38i1.27774},
}
Figure