AImoclips: A Benchmark for Evaluating Emotion Conveyance in Text-to-Music Generation

Gyehun Go, Satbyul Han, Ahyeon Choi, Eunjin Choi, Juhan Nam, Jeong Mi ParkarXiv 2025 · DOI · arXiv

Introduction

As text-to-music (TTM) generation reaches new heights in audio fidelity, a critical question remains: does the music actually convey the intended emotion? While existing evaluations focus on technical quality or prompt alignment (e.g., "is there a piano?"), they often overlook the psychological impact on the listener. The AImoclips benchmark addresses this gap by quantifying emotion conveyance—the alignment between the emotional intent of a text prompt and the human perception of the resulting audio.

Methodology

The researchers developed a rigorous framework to evaluate six state-of-the-art models: four open-source (AudioLDM 2, MusicGen, Mustango, Stable Audio Open) and two commercial (Suno v4.5, Udio v1.5). The process involved four key stages:

  1. Emotion Selection: 12 emotion words were selected to represent the four quadrants of the Valence-Arousal (VA) model. Valence measures pleasure (positive vs. negative), while Arousal measures energy (high vs. low).
  2. Generation: The models generated 991 music clips (10 seconds each) based on these emotional prompts.
  3. Human Rating: 111 participants rated the clips on a 9-point scale for both valence and arousal.
  4. Analysis: The team established ground truth scores using the Warriner et al. (ANEW extension) dictionary to measure how far the AI-generated music deviated from the intended emotional weight.
Visual representation of the 9-point Valence-Arousal (VA) scale used by human raters.

Key Results and Insights

The "Neutrality Bias"

A primary finding of the study is the "centralizing tendency" or neutrality bias. Regardless of how extreme the emotional prompt was, models tended to produce music that listeners rated as more emotionally neutral. As shown in the quadrant analysis, high-valence prompts often resulted in lower-than-intended pleasure ratings, while low-valence prompts were perceived as less negative than intended.

Mean score deviations across emotional quadrants, highlighting the 'neutrality bias' and the performance gap between valence and arousal.

Arousal vs. Valence

The study revealed that AI models are significantly better at conveying energy (arousal) than specific moods (valence). It is computationally simpler for a model to associate "energetic" with acoustic features like high tempo and volume than it is to distinguish the subtle musical nuances between "calm" (positive/low energy) and "gloomy" (negative/low energy).

Commercial vs. Open-Source Performance

A clear divide emerged between model types. Commercial models (Suno, Udio) exhibited a positive valence bias, often sounding more pleasant than intended. The authors suggest that the high production quality and "catchiness" of these models may inherently bias listeners toward positive feelings. Conversely, open-source models tended to lean toward the unpleasant (negative valence) side of the spectrum.

Conclusion and Future Work

AImoclips highlights a significant limitation in current TTM systems: a lack of "affective controllability." While models can follow instrumental instructions, they struggle to hit specific psychological targets reliably. Future work will focus on identifying the specific acoustic features (tempo, key, spectral brightness) that drive these emotional biases. The ultimate goal is to leverage the AImoclips dataset to fine-tune models, moving toward "emotion-to-music" systems capable of nuanced, intentional emotional expression.

Summary updated

Abstract

Recent advances in text-to-music (TTM) generation have enabled controllable and expressive music creation using natural language prompts. However, the emotional fidelity of TTM systems remains largely underexplored compared to human preference or text alignment. In this study, we introduce AImoclips, a benchmark for evaluating how well TTM systems convey intended emotions to human listeners, covering both open-source and commercial models. We selected 12 emotion intents spanning four quadrants of the valence-arousal space, and used six state-of-the-art TTM systems to generate over 1,000 music clips. A total of 111 participants rated the perceived valence and arousal of each clip on a 9-point Likert scale. Our results show that commercial systems tend to produce music perceived as more pleasant than intended, while open-source systems tend to perform the opposite. Emotions are more accurately conveyed under high-arousal conditions across all models. Additionally, all systems exhibit a bias toward emotional neutrality, highlighting a key limitation in affective controllability. This benchmark offers valuable insights into model-specific emotion rendering characteristics and supports future development of emotionally aligned TTM systems.
BibTeX
@misc{go2025aimoclips,
  author = {Gyehun Go and Satbyul Han and Ahyeon Choi and Eunjin Choi and Juhan Nam and Jeong Mi Park},
  title = {{AImoclips: A Benchmark for Evaluating Emotion Conveyance in Text-to-Music Generation}},
  year = {2025},
  eprint = {2509.00813},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2509.00813},
}
Figure