Predicting Perceived Music Emotions with Respect to Instrument Combinations

Viet Dung Nguyen, Quan H. Nguyen, Richard G. FreedmanAAAI 2023 · DOI

Introduction

Music Emotion Recognition (MER) is a critical task for modern music recommendation systems and visualization tools. While human emotional perception is deeply tied to timbre—the unique quality of different instruments—most AI models process audio as a single, combined signal. This research investigates whether training models on individual instrument tracks (stems) improves their ability to perceive emotion compared to training on the full original mix.

Methodology

The study compared two distinct machine learning approaches to evaluate the impact of source separation on emotion recognition:

  1. Convolution Recurrent Neural Network (CRNN): This deep learning model employs convolutional layers to extract spectral features from audio spectrograms, followed by Long Short-term Memory (LSTM) blocks to capture temporal dynamics. The specific architecture is detailed in Figure 1.
  2. Random Forest (RF): A baseline ensemble model using over 6,000 acoustic features extracted via the OpenSMILE toolkit.
The CRNN architecture used for music emotion recognition, featuring convolutional layers, LSTM blocks, and dense layers for final classification.

To test the core hypothesis, the researchers used a pre-trained Wave-U-Net model to de-mix songs from the PMEmo dataset into four stems: vocals, bass, drums, and "others." Models were then trained on two datasets: the original "Mixed" audio and the "Sep" (separated) instrumental tracks. Performance was measured using Kullback–Leibler (KL) Divergence to compare predicted emotion distributions across valence and arousal dimensions against human ground-truth ratings.

Results and Discussion

Contrary to the initial hypothesis, the results demonstrated that the "whole is greater than the sum of its parts." For the CRNN model, training on the original mixed audio consistently yielded higher accuracy (lower KL Divergence) than training on separated tracks.

As shown in the performance data in Figure 6, the CRNN trained on mixed data ($E_{\text{mixed}}$) outperformed the separated version ($E_{\text{sep}}$) in the vast majority of cross-validation iterations. Statistical analysis confirmed that the mixed-data model was significantly more accurate in 7 out of 10 iterations ($p < 0.01$).

Comparative performance metrics across 10 iterations, showing KL divergence for mixed ($E_{\text{mixed}}$) versus separated ($E_{\text{sep}}$) audio sources.

This "Mixed Advantage" suggests that the complex interactions between instruments—such as harmony and blending—provide vital emotional cues that are lost during isolation. Interestingly, the Random Forest baseline showed negligible differences between the two data types, indicating that deep learning architectures like CRNNs are far more sensitive to the structural integrity of the audio signal.

Conclusion and Future Work

The study concludes that current source separation techniques do not necessarily enhance music emotion recognition; in fact, they may hinder a neural network's ability to grasp emotional context. The researchers identified several limiting factors, including the "domain gap" between the source separation training data (MUSDB18) and the emotion dataset (PMEmo), as well as the simplified categorization of instruments into only four stems. Future research will focus on more granular instrument separation and introspective analysis of specific musical elements that AI still finds challenging to interpret.

Summary updated

BibTeX
@inproceedings{nguyen2023predicting,
  author = {Viet Dung Nguyen and Quan H. Nguyen and Richard G. Freedman},
  title = {{Predicting Perceived Music Emotions with Respect to Instrument Combinations}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2023},
  doi = {10.1609/aaai.v37i13.26910},
}
Figure