Abstract
This study explores the efficacy of deep learning models in predicting musical mood from both audio signals and lyrical content. By comparing traditional feature-engineering approaches with various neural network architectures, researchers from Deezer demonstrate that deep learning particularly excels in predicting "arousal" (energy). Furthermore, the study introduces a "mid-level fusion" strategy that integrates audio and lyric features before the final output layer, yielding significant improvements in "valence" (positivity) prediction. These findings underscore the importance of modeling the complex interplay between musical sound and poetic text for accurate emotion detection.
Introduction
Music mood detection is a fundamental component of modern recommendation systems, yet it remains a challenging task due to the subjective and multimodal nature of emotion. Traditionally, this field has relied on hand-crafted features, such as Mel-Frequency Cepstral Coefficients (MFCCs) for audio and sentiment lexicons for lyrics. However, these methods often struggle to capture the non-linear relationships inherent in music. This research investigates whether end-to-end deep learning can outperform classical methods and identifies the most effective strategies for multimodal data fusion.
Methodology
The researchers evaluated several architectures across three modalities: audio, lyrics, and bimodal fusion.
- Audio Processing: A Convolutional Neural Network (ConvNet) was employed to process mel-spectrograms.
- Lyric Processing: Various Recurrent Neural Network (RNN) architectures, including GRUs and LSTMs, were tested using Word2Vec embeddings.
- Fusion Strategies: The core contribution involves comparing "Late Fusion" (averaging unimodal predictions) with "Mid-level Fusion" (middleDL). In the latter, internal representation layers from the audio and lyric networks are concatenated, allowing the model to learn cross-modal correlations.
The study utilized a large-scale dataset of 18,644 tracks, mapping metadata between the Million Song Dataset and the Deezer catalog. Ground truth labels for valence and arousal were derived from user-generated LastFM tags mapped to a psychological lexicon.
Key Results
The experimental results, measured by the $R^2$ coefficient, indicate that deep learning models generally match or exceed the performance of classical approaches (CA).
Key findings include:
- Arousal is Audio-Driven: Arousal prediction is dominated by the audio signal, with the ConvNet achieving an $R^2$ of 0.235. Lyrical content provided negligible gains for this dimension.
- Valence requires Multimodality: Predicting valence is more complex, as lyrics and audio often provide complementary or even conflicting emotional cues. The mid-level fusion model (
middleDL) achieved an $R^2$ of 0.219 for valence, outperforming the late fusion approach ($R^2 = 0.194$). - Classical Lexicons remain Competitive: Interestingly, traditional models using expert-curated sentiment lexicons performed strongly on lyrics-only valence, suggesting that deep learning models still have room to improve in learning sentiment from scratch.
Discussion and Conclusion
The study confirms that deep learning can effectively automate music mood detection without manual feature engineering. The success of mid-level fusion highlights the necessity of capturing the interaction between lyrics and audio. Future work will focus on improving temporal alignment between text and sound, reducing label noise from social tags, and exploring unsupervised pre-training to further enhance model robustness.
Summary updated