Music Emotion Recognition by Multi-label Multi-layer Multi-instance Multi-view Learning

Bin Wu, Erheng Zhong, Andrew Horner, Qiang Yang2014 · DOI

TL;DR

This paper tackles the challenge of accurately recognizing music emotions by proposing a novel Hierarchical Music Emotion Recognition (HMER) model. HMER leverages a multi-layer, multi-instance, multi-view learning approach, treating music as a hierarchical structure of songs, segments, and sentences, and incorporating both music and lyrics as distinct "views." This allows for a more nuanced understanding of how emotions evolve within a song, overcoming the limitations of previous song-level analysis.

What Problem Does This Paper Address?

Music emotion recognition (MER) is crucial for applications like music search and recommendation. However, current MER systems have hit a performance plateau. A key reason is that music emotions aren't static; they change over time within a song. Traditional methods often treat an entire song as a single data point, averaging out these emotional shifts and failing to capture the intricate dynamics. This paper addresses this by recognizing that emotions can vary significantly between song segments and even sentences, and that existing methods don't adequately model this hierarchical and dynamic nature of music emotion.

Methods

The core of this paper is the Hierarchical Music Emotion Recognition (HMER) model. HMER is a novel hierarchical Bayesian model that frames MER as a multi-label, multi-layer, multi-instance, multi-view learning problem.

  • Multi-label: A song can have multiple emotions associated with it.
  • Multi-layer Multi-instance: Music is structured hierarchically (song -> segment -> sentence), and labels can apply to any of these instances. This is a key innovation, moving beyond single-layer multi-instance learning.
  • Multi-view: HMER utilizes two "views" of the music: acoustic features (music) and lyrical content. These views are synchronized at the sentence level.

HMER models music emotion dynamics by assuming:

  1. Emotions vary greatly between segments.
  2. Emotions are mostly consistent within each segment.
  3. Emotions are almost always consistent within each sentence.

The model uses sentence-level music and lyrics features and captures emotion correlations between segments through a Markov process, and stricter correlations within segments by assuming sentences follow the same emotion distribution.

Datasets

The authors constructed their dataset by:

  • Utilizing the All Music Guide for emotion labels (183 emotion labels, 5106 label-song entries).
  • Linking songs with the Million Song Dataset for acoustic feature extraction.
  • Downloading LRC lyrics files for sentence-level synchronization.

The final dataset comprised 1493 songs and 122 labels.

Key Insights and Ideas

  • Hierarchical Structure is Key: The most significant insight is that music emotion is not monolithic. Modeling it as a hierarchy of song, segment, and sentence allows for a much richer and more accurate representation of emotional dynamics.
  • Sentence-Level Synchronization Matters: Leveraging sentence-level synchronization between music and lyrics is crucial for capturing fine-grained emotional nuances, something that song-level or even segment-level synchronization might miss.
  • Emotion Correlations are Multi-faceted: HMER's ability to model correlations between segments (via a Markov process) and within segments (sentences sharing distributions) is a critical advancement over previous methods that treated instances as exchangeable.
  • Multi-view Learning Enhances Understanding: Combining acoustic and lyrical information provides a more comprehensive picture of the music's emotional content than either modality alone.

Key Results

HMER significantly outperformed several state-of-the-art methods, including single-instance models (BR, LP, RAKEL) and existing multi-instance models (M3LDA-seg, M3LDA-sen).

  • F₁ Score: HMER achieved the best F₁ score, showing a substantial improvement over baselines. For instance, it was 49% better than the best existing multi-label music emotion recognition baseline (RAKEL) and 7.9% better than M3LDA.
  • Mean Average Precision (MAP@10): HMER also led in MAP@10, indicating better ranking of relevant songs for emotion queries. It showed a 326% improvement over RAKEL and an 8.0% improvement over M3LDA.
  • HMER vs. HMER-E: The version of HMER that optimizes the parameter $\xi$ (capturing segment correlations) performed better than the version that kept it fixed, demonstrating the value of modeling inter-segment emotion correlations.
  • Computational Complexity: While HMER is computationally more intensive than single-instance methods, its performance gains justify its use for offline processing.

Limitations and Future Work

  • Computational Cost: The multi-layer, multi-instance, multi-view nature of HMER leads to higher computational complexity, making it primarily suitable for offline tasks rather than real-time applications.
  • Future Work: The authors suggest exploring parallel implementation of HMER to speed up training. They also propose extending the model to incorporate more "views," such as video clips from music videos, to further enhance emotion recognition. Additionally, they mention the potential for improved performance with more advanced feature extraction and selection techniques.

Summary updated

Abstract

Music emotion recognition, which aims to automatically recognize the affective content of a piece of music, has become one of the key components of music searching, exploring, and social networking applications. Although researchers have given more and more attention to music emotion recognition studies, the recognition performance has come to a bottleneck in recent years. One major reason is that experts' labels for music emotion are mostly song-level, while music emotion usually varies within a song. Traditional methods have considered each song as a single instance and have built models based on song-level features. However, they ignored the dynamics of music emotion and failed to capture accurate emotion-feature correlations. In this paper, we model music emotion recognition as a novel multi-label multi-layer multi-instance multi-view learning problem: music is formulated as a hierarchical multi-instance structure (e.g., song-segment-sentence) where multiple emotion labels correspond to at least one of the instances with multiple views of each layer. We propose a Hierarchical Music Emotion Recognition model (HMER) -- a novel hierarchical Bayesian model using sentence-level music and lyrics features. It captures music emotion dynamics with a song-segment-sentence hierarchical structure. HMER also considers emotion correlations between both music segments and sentences. Experimental results show that HMER outperforms several state-of-the-art methods in terms of $F_1$ score and mean average precision.
BibTeX
@misc{wu2014music,
  author = {Bin Wu and Erheng Zhong and Andrew Horner and Qiang Yang},
  title = {{Music Emotion Recognition by Multi-label Multi-layer Multi-instance Multi-view Learning}},
  year = {2014},
  doi = {10.1145/2647868.2654904},
}
Figure