MERP: A Music Dataset with Emotion Ratings and Raters' Profile Information

Enyan Koh, Kin Wai Cheuk, Kwan Yee Heung, Kat R. Agres, Dorien HerremansSensors 2023 · DOI · OpenReview

TL;DR

This paper introduces MERP, a novel music dataset featuring dynamic emotion ratings (valence and arousal) for full-length songs, crucially augmented with detailed rater profile information. This dataset aims to address the subjectivity in music emotion recognition by exploring how individual differences influence perception, and provides baseline models to facilitate future research in personalized affective computing.

What Problem Does This Paper Address?

The automatic recognition of emotions in music (Music Emotion Recognition or MER) is a growing field, but it faces a significant challenge: the inherent subjectivity of emotional perception. Listeners often have differing views on the emotions conveyed by a piece of music, leading to noisy labels in existing datasets. While some datasets offer dynamic (time-varying) emotion annotations, they are limited in size and scope. Furthermore, the influence of individual listener characteristics (like musical background, age, or culture) on emotion perception is not well-understood or systematically captured in datasets. This paper addresses these gaps by creating a large, publicly available dataset that not only provides rich, dynamic emotion labels but also includes detailed demographic and preference data from the raters, enabling the study of personalized MER.

Methods

The MERP dataset was collected using Amazon Mechanical Turk (MTurk), a crowdsourcing platform. To ensure data quality, a multi-stage filtering process was employed. This included:

  • Initial Screening: Removing participants who did not complete the task or submitted conflicting information.
  • Abnormal Length Filtering: Discarding trials with significantly deviating lengths from expected sampling rates.
  • Benchmark Comparison: Comparing participant ratings against pre-labeled benchmark songs from the DEAM dataset, removing participants whose ratings deviated significantly.

For emotion prediction, two baseline models were developed:

  • Fully Connected Model: A dense neural network designed for straightforward prediction.
  • LSTM Model: A recurrent neural network architecture chosen for its ability to handle sequential data like music.

Both models were trained using audio features extracted with the openSMILE toolbox and, in a separate experiment, audio features concatenated with individual rater profile features.

Datasets

  • MERP Dataset (New): This is the primary contribution of the paper. It comprises 50 full-length songs selected from the Free Music Archive (FMA) using an innovative method to ensure diverse emotional representation across the valence-arousal space. Additionally, 4 song excerpts from the DEAM dataset were included as benchmarks. The dataset contains dynamic valence and arousal ratings from 277 participants, along with their demographic, listening preference, and musical background information.
  • DEAM Dataset: Used for quality control and benchmarking.

Key Insights and Ideas

  • Subjectivity is Key: The paper strongly emphasizes that emotion perception in music is highly subjective, and this subjectivity is a primary driver of noise in MER datasets.
  • Profile Matters: Statistical analysis revealed significant differences in emotion ratings based on various rater profile features (e.g., age, country of residence, musical enculturation, genre preference, musical training). This suggests that individual differences are not just noise but can be informative for understanding and potentially improving MER.
  • Dynamic Ratings are Crucial: The use of dynamic, continuous valence and arousal ratings throughout full-length songs captures the temporal evolution of emotion in music, offering a richer representation than static labels.
  • Crowdsourcing Quality Control: The paper demonstrates the effectiveness of a multi-stage filtering process, including benchmark comparisons, for obtaining high-quality subjective data from crowdsourcing platforms like MTurk.

Key Results

  • Dataset Size: The MERP dataset contains 277 participants and 3482 trials after preprocessing.
  • Profile Influence: Statistical tests (Kruskal Wallis and Dunn's tests) showed significant differences in valence and arousal ratings across various profile groups. For instance, age, country of residence, and musical enculturation were found to significantly influence ratings.
  • Model Performance (Audio Only): The fully connected model outperformed the LSTM model in terms of Mean Squared Error (MSE) for both valence and arousal when using audio features alone. This suggests that for this dataset and feature set, a simpler model might be sufficient.
  • Model Performance (with Profile Info): Incorporating profile features generally improved prediction performance, particularly for the fully connected model. The paper highlights that models including features related to musical background (like instrument playing or training) showed promising results.

Limitations and Future Work

  • Profile Binning: The authors acknowledge that the binning of non-boolean profile features (like age or genre) was arbitrary and suggest that future research could explore different configurations for more representative bins.
  • Limited Genre Representation: Some preferred genres were underrepresented in the dataset, potentially limiting the ability to draw strong conclusions about genre-specific emotion perception.
  • Model Simplicity: The baseline models (fully connected and LSTM) are intentionally simple for benchmarking. Future work could explore more advanced architectures like CNNs or Transformers.
  • Dataset Expansion: The authors suggest expanding MERP by including more participants from diverse countries, adding more royalty-free music, and potentially incorporating other emotion models or features.
  • Investigating Musical Features: The dataset provides an avenue for researchers to further examine how specific musical features influence emotion perception across different listener groups.
  • Subjectivity Handling: Future research could focus on developing models that explicitly account for individual subjectivity and personalize emotion predictions.

Summary updated

BibTeX
@article{koh2023merp,
  author = {Enyan Koh and Kin Wai Cheuk and Kwan Yee Heung and Kat R. Agres and Dorien Herremans},
  title = {{MERP: A Music Dataset with Emotion Ratings and Raters' Profile Information}},
  journal = {Sensors},
  year = {2023},
  doi = {10.3390/s23010382},
}
Figure