SongEval: A Benchmark Dataset for Song Aesthetics Evaluation

Jixun Yao, Guobin Ma, Huixin Xue, Huakang Chen, Chunbo Hao, Yuepeng Jiang, Haohe Liu, Ruibin Yuan, Jin Xu, Wei Xue, Lei XieSubmission to ICLR 2026 · Withdrawn

TL;DR

Evaluating AI-generated songs using traditional signal- or embedding-based metrics fails to capture subjective qualities like musicality, vocal phrasing, or song structure. To bridge this gap, the authors present SongEval, the first large-scale, open-source benchmark featuring over 140 hours (2,399 full-length tracks) of generated music scored by professional musicians across five aesthetic dimensions. Models trained on SongEval align significantly closer to human musical judgment than existing objective audio quality metrics.

What Problem Does This Paper Address?

Recent advances in generative AI (like Suno, Udio, DiffRhythm, and YuE) can produce complete, multi-minute songs featuring both vocals and instrumental accompaniment. However, evaluating these models remains a major bottleneck. Standard objective metrics—such as mel-spectrogram distances, pitch tracking, or acoustic embedding distances—measure acoustic fidelity rather than musical appeal, coherence, or emotional resonance.

While the speech domain has long used Mean Opinion Score (MOS) benchmarks to evaluate synthesized voices, the music domain lacks full-length song evaluation datasets with fine-grained aesthetic dimensions. SongEval addresses this gap by creating an expert-annotated benchmark for full-length song aesthetics.

Methods

The authors built SongEval using a two-stage data generation and expert annotation pipeline:

  1. Prompt & Lyric Generation: ChatGPT generated genre-aligned prompts (describing style, instrumentation, and mood) and matching lyrics in English and Mandarin across nine mainstream genres.
  2. Song Generation & Filtering: Five generative systems—Suno, Udio, Mureka, YuE, and DiffRhythm—synthesized full-length tracks. The authors applied vocal-range filtering to remove non-singing or instrumental-only failures.
  3. Professional Annotation: Sixteen trained musicians evaluated each track on a 1–5 scale across five core dimensions:
  4. Overall Coherence (musical and emotional continuity)
  5. Memorability (catchiness and motifs)
  6. Naturalness of Vocal Breathing and Phrasing
  7. Clarity of Song Structure (verses, choruses, bridges)
  8. Overall Musicality (listening enjoyment, mix, and harmony) Each song was independently rated by four musicians.

To automate aesthetic scoring, the authors adapted four predictive architectures:

  • MOSNet-based (CNN + BLSTM)
  • LDNet-based (listener-dependent modeling)
  • SSL-based (using the self-supervised music representation model MuQ with a regression head)
  • UTMOS-based (an ensemble model combining strong and weak learners)

Datasets

  • SongEval (Introduced): Contains 2,399 full-length songs (over 140 hours; ~3.5 minutes average duration) spanning English (EN) and Chinese (ZH), 9 musical genres (Pop, Rock, Electronic, Blues, Jazz, Classical, Hip-hop/Rap, Country, World Music), and an "Others" category containing real non-copyrighted songs and problematic edge cases.
  • Comparison with Prior Benchmarks: Unlike MusicEval (16.7 hours, accompaniment-only, no vocal evaluations) or AES-Natural (29.4 hours, short clips across generic audio/speech/music), SongEval focuses entirely on full-length complete songs with vocals across five specialized musical dimensions.

Key Insights and Ideas

  • Pretrained Representations Matter: SSL-based and UTMOS-based architectures substantially outperformed traditional MOS models (MOSNet and LDNet). Leveraging self-supervised representations (like MuQ) allows models to perceive long-range musical structures and nuanced audio interactions rather than overfitting to low-level acoustic cues.
  • Objective Audio Metrics Fail at Semantics: Existing objective metrics (e.g., Audiobox-Aesthetic metrics like Production Quality and Content Enjoyment) correlate poorly with human ratings on dimensions like memorability and coherence. Dedicated subjective aesthetic supervision is essential to evaluate musical artistry.

Key Results

  • Predictive Accuracy:
  • On utterance-level evaluation, the UTMOS-based model achieved the strongest correlations with human scores: Linear Correlation Coefficient (LCC) of 0.908–0.917 and Spearman Rank Correlation (SRCC) of 0.894–0.901 across all five dimensions.
  • On system-level evaluation, UTMOS and SSL models achieved LCC values exceeding 0.95–0.96.
  • Comparison to Existing Metrics:
  • Evaluating Pearson correlation against human ground truth, the UTMOS model achieved an average correlation of 0.912.
  • In comparison, standard Audiobox-Aesthetic metrics lagged significantly: Content Enjoyment (0.614), Content Usefulness (0.662), Production Complexity (0.408), Production Quality (0.630), and Vocal Range (0.702).

Limitations and Future Work

  • Perceptual Entanglement: The five dimensions inevitably exhibit some subjective correlation (e.g., structural clarity closely influences perceived coherence, and musicality overlaps with memorability).
  • Future Directions: The authors aim to build more granular, robust automatic evaluation models capable of generalizing across even wider varieties of musical styles, sub-genres, and languages, as well as applying SongEval scores to guide controllable music generation.

Summary updated

Abstract

Aesthetics serve as an implicit and important criterion in song generation tasks that reflect human perception beyond objective metrics. However, evaluating the aesthetics of generated songs remains a fundamental challenge, as the appreciation of music is highly subjective. Existing evaluation metrics, such as embedding-based distances, are limited in reflecting the subjective and perceptual aspects that define musical appeal. To address this issue, we introduce SongEval, the first open-source, large-scale benchmark dataset for evaluating the aesthetics of full-length songs. SongEval includes over 2,399 songs in full length, summing up to more than 140 hours, with aesthetic ratings from 16 professional annotators with musical backgrounds. Each song is evaluated across five key dimensions: overall coherence, memorability, naturalness of vocal breathing and phrasing, clarity of song structure, and overall musicality. The dataset covers both English and Chinese songs, spanning nine mainstream genres. Moreover, to assess the effectiveness of song aesthetic evaluation, we conduct experiments using SongEval to predict aesthetic scores and demonstrate better performance than existing objective evaluation metrics in predicting human-perceived musical quality.
BibTeX
@misc{yao2025songeval,
  author = {Jixun Yao and Guobin Ma and Huixin Xue and Huakang Chen and Chunbo Hao and Yuepeng Jiang and Haohe Liu and Ruibin Yuan and Jin Xu and Wei Xue and Hao Liu and Lei Xie},
  title = {{SongEval: A Benchmark Dataset for Song Aesthetics Evaluation}},
  year = {2025},
  eprint = {2505.10793},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2505.10793},
  howpublished = {arXiv},
}
Figure