Music Emotion Recognition via End-to-End Multimodal Neural Networks

Byungsoo Jeon, Chanju Kim, Adrian Kim, Dongwon Kim, Jangyeon Park, JungWoo HaRecSys Posters 2017 · OpenReview

TL;DR

This paper introduces an end-to-end multimodal neural network for music emotion recognition (MER) that leverages both acoustic signals and lyrics. By avoiding manual feature engineering and training on a new, high-quality K-pop dataset, they achieve a state-of-the-art accuracy of 80% for classifying music as positive or negative. This work is crucial for improving context-aware music recommendation systems.

What Problem Does This Paper Address?

Music emotion recognition (MER) is a critical component for building intelligent music recommendation systems that can cater to a user's current mood. While it's understood that factors like tempo, tone, and lyrical content influence a song's emotional impact, accurately capturing this relationship has been challenging. Existing MER methods often rely on labor-intensive, hand-crafted features extracted from audio and lyrics, which can be suboptimal and time-consuming. The authors aim to bridge this gap by developing a more automated, end-to-end approach that directly learns from raw audio and lyrical data. They simplify the problem to a binary classification (positive/negative emotion) to make it more practical for recommendation scenarios.

Methods

The core of this paper is a novel end-to-end multimodal neural network architecture. This model comprises two parallel branches: one for processing acoustic signals and another for lyrics.

  • Audio Branch: Mel-spectrograms are fed into a series of 1D convolutional and max-pooling layers to extract hierarchical features. These features are then processed by two GRU (Gated Recurrent Unit) layers, followed by a fully connected layer, to generate a fixed-length audio embedding.
  • Lyrics Branch: Word embeddings are first generated from lyrics. These are then passed through a 1D convolutional layer and a global 1D max-pooling layer to capture salient lyrical patterns. Finally, a fully connected layer produces a fixed-length lyrics embedding.

Crucially, these two branches are concatenated, and the combined representation is fed into another fully connected layer with a softmax activation to produce the final binary emotion classification. The entire network is trained end-to-end, meaning it learns to extract relevant features from both modalities simultaneously without requiring pre-defined feature engineering.

Datasets

The authors introduce a new dataset specifically curated for this research. It consists of approximately 7,000 K-pop tracks, with an equal split of positive and negative emotions. The dataset was sourced from a Korean music streaming service (NAVER Music) and includes both audio (represented as mel-spectrograms) and lyrical content. Tracks were labeled as positive or negative based on an editor-defined emotion word dictionary, filtering out ambiguous cases. Tracks shorter than one minute or with fewer than 30 words in their lyrics were excluded. For evaluation, the dataset was split into 90% for training and 10% for validation.

Key Insights and Ideas

  • End-to-End Learning is Powerful: The paper demonstrates that an end-to-end approach, which learns features directly from raw data, significantly outperforms methods relying on manual feature engineering for MER.
  • Lyrics Hold Significant Emotional Weight: The results indicate that lyrical content, when processed effectively, can be a stronger predictor of music emotion than acoustic signals alone. This highlights the importance of semantic understanding in MER.
  • Multimodal Fusion is Key: Combining information from both audio and lyrics yields the best performance, suggesting that a comprehensive understanding of a song's emotional impact requires considering multiple modalities.
  • Simplified Binary Classification is Effective: For practical recommendation systems, a binary (positive/negative) emotion classification is sufficient and achievable with high accuracy, reducing the complexity of dealing with numerous emotion categories.

Key Results

The multimodal model achieved a peak accuracy of 80.46% on the K-pop dataset. This significantly outperformed unimodal models:

  • The best unimodal audio model (CNN+RNN) achieved 66.19% accuracy.
  • The best unimodal lyrics model (1D CNN) achieved 78.15% accuracy. The paper also notes that the lyrics-only model performed better than the audio-only model, reinforcing the insight about the importance of lyrics. Validation curves showed that the lyrics and multimodal models had more stable convergence than the audio-only model.

Limitations and Future Work

The authors acknowledge that their current work focuses on a binary classification of emotion (positive/negative) and a specific genre (K-pop). They suggest future work could involve:

  • Investigating end-to-end deep learning strategies with a wider range of emotional categories.
  • Expanding the dataset to include more diverse genres and a larger number of tracks.
  • Applying the developed multimodal MER method to real-world context-aware music recommendation services, such as Clova, an AI assistant platform.

Summary updated

BibTeX
@inproceedings{jeon2017music,
  author = {Byungsoo Jeon and Chanju Kim and Adrian Kim and Dongwon Kim and Jangyeon Park and JungWoo Ha},
  title = {{Music Emotion Recognition via End-to-End Multimodal Neural Networks}},
  booktitle = {RecSys Posters},
  year = {2017},
  url = {https://openreview.net/forum?id=rcWAhKlA16g},
}
Figure