MusicSem: A Semantically Rich Language--Audio Dataset of Natural Music Descriptions

Rebecca Salganik, Teng Tu, Fei-Yueh Chen, Xiaohao Liu, Keifeng Lu, Ethan Luvisia, Zhiyao Duan, Guillaume Salha-Galvan, Anson Kahng, Yunshan Ma, Jian Kang

TL;DR

Existing multimodal music models excel at matching technical descriptors like "guitar" or "fast tempo," but often fail to grasp natural human context like "songs for a rainy day drive." To bridge this gap, the authors introduce MusicSem, a dataset of 32,493 language-audio pairs scraped from organic Reddit discussions. By structuring these natural descriptions into a five-part semantic taxonomy, MusicSem exposes the blind spots of current models and provides a powerful new benchmark for human-aligned music AI.

What Problem Does This Problem Address?

Multimodal music representation learning underpins tasks like cross-modal retrieval (text-to-music search), text-to-music generation, and automated music captioning. However, current state-of-the-art models suffer from a "semantic blind spot." They are typically trained on datasets annotated by professional musicians—which focus heavily on technical attributes like instrumentation, genre, and tempo—or on synthetic datasets generated by LLMs.

Consequently, when an everyday listener describes music using abstract, emotional, or situational language (e.g., "a song that feels like a warm hug on a cold day"), models struggle to capture the underlying intent. The field lacks large-scale, grounded datasets that reflect how humans naturally contextualize their listening experiences.

Methods

The authors built a rigorous pipeline to transform messy, organic social media discourse into a structured, high-quality dataset:

  1. Thread Selection: Scraped posts from 2008 to 2022 across five popular music-focused subreddits (such as r/popheads and r/musicsuggestions) using the Pushshift API.
  2. Information Extraction: Prompted GPT-4o to identify (song, artist) pairs and categorize text snippets into five distinct semantic categories.
  3. Validation & Grounding: Filtered out low-confidence extractions using a character-overlap threshold (>75%) and resolved ambiguities via the Spotify API to fetch unique track IDs. Audio files were retrieved using spotdl.
  4. Hallucination Mitigation: Employed Claude 3.7 Sonnet as an independent verifier to check if the LLM-generated sentence-level summaries matched the original raw text, followed by manual verification by human experts (including trained musicians).

Datasets

  • MusicSem (New): Contains 32,493 language-audio pairs covering 11,842 unique songs and 4,430 unique artists. It features a massive vocabulary expansion (22,738 distinct words) compared to older benchmarks.
  • Baseline Benchmarks: The paper evaluates and compares MusicSem against canonical human-annotated datasets: MusicCaps (5,521 pairs) and Song Describer (1,106 pairs).

As shown in the dataset comparison table, MusicSem offers vastly superior coverage of non-descriptive semantics:

Table 1: Statistics of existing language–audio music datasets that are most comparable to our setting. L–A Pairs denotes the number of language–audio pairs, while Annotation indicates the origin of the textual annotations describing the music.

Key Insights and Ideas

  • The "Surface-Level" Bias: Current text-to-audio generative and retrieval models are highly sensitive to descriptive changes (e.g., changing "piano" to "guitar") but virtually blind to situational or atmospheric shifts (e.g., changing "listening at church" to "during a psychedelic experience").
  • Dataset-Induced Rankings: A model's performance ranking changes completely depending on the evaluation dataset. For instance, CLAP dominates on MusicCaps (which contains ambient noise and YouTube clips), while CLaMP3 wins on Song Describer and MusicSem (which feature studio-quality tracks). This highlights severe generalization limits in current architectures.
  • The Illusion of High Scores: In music-to-text generation (captioning), higher objective NLP metrics (like BLEU or CIDEr) do not guarantee semantic accuracy. Qualitative analysis showed that models often achieve decent scores through superficial lexical overlap while completely hallucinating or misidentifying key musical attributes.

Key Results

  • Semantic Diversity: While MusicCaps and Song Describer feature contextual tags in fewer than 10% of their entries, 77% of MusicSem's entries include contextual semantics, alongside high coverage for atmospheric (64%) and situational (48%) contexts.
  • Retrieval & Generative Sensitivity: Baseline evaluations on text-to-audio retrieval and generation models confirmed their poor semantic awareness. For example, in retrieval tasks ($k=10$), models like LARP scored 0.0 on metadata ($R^m$) and a meager 0.06 on situational context ($R^s$).
Semantic sensitivity analysis of generative and retrieval models
  • Fine-Tuning Impact: To prove MusicSem's value as a training resource, the authors fine-tuned a CLAP model on it. The results were dramatic: Recall@1 surged by +450% (from 0.82 to 4.51) and Mean Reciprocal Rank (MRR) improved by +96.77%. Crucially, its semantic sensitivity to contextual and situational prompts improved by +48.28% and +28.13%, respectively.

Limitations and Future Work

  • Entity Disambiguation: Informal text makes precise text-to-song alignment difficult. Ambiguous references (e.g., "Boston" could mean the band, an album, or a song title) remain an open Named Entity Recognition (NER) challenge in the music domain.
  • Demographic Bias: Sourcing data exclusively from English-language subreddits introduces inherent selection biases, skewing toward younger, Western, and digitally engaged populations.
  • Future Directions: The authors plan to expand the dataset's scale, incorporate lyrics and symbolic musical representations, and explore evaluations for controllable music generation and conversational music recommendation agents.

Summary updated

Abstract

Music representation learning is central to music information retrieval and generation. While recent advances in multimodal learning have improved alignment between text and audio for tasks such as cross-modal music retrieval, text-to-music generation, and music-to-text generation, existing models often struggle to capture users' expressed intent in natural language descriptions of music. This observation suggests that the datasets used to train and evaluate these models do not fully reflect the broader and more natural forms of human discourse through which music is described. In this paper, we introduce MusicSem, a dataset of 32,493 language-audio pairs derived from organic music-related discussions on the social media platform Reddit. Compared to existing datasets, MusicSem captures a broader spectrum of musical semantics, reflecting how listeners naturally describe music in nuanced and human-centered ways. To structure these expressions, we propose a taxonomy of five semantic categories: descriptive, atmospheric, situational, metadata-related, and contextual. In addition to the construction, analysis, and release of MusicSem, we use the dataset to evaluate a wide range of multimodal models for retrieval and generation, highlighting the importance of modeling fine-grained semantics. Overall, MusicSem serves as a novel semantics-aware resource to support future research on human-aligned multimodal music representation learning.
BibTeX
@inproceedings{salganik2025musicsem,
  author = {Rebecca Salganik and Teng Tu and Fei-Yueh Chen and Xiaohao Liu and Keifeng Lu and Ethan Luvisia and Zhiyao Duan and Guillaume Salha-Galvan and Anson Kahng and Yunshan Ma and Jian Kang},
  title = {{MusicSem: A Semantically Rich Language--Audio Dataset of Natural Music Descriptions}},
  booktitle = {AI4Music @ NeurIPS},
  year = {2025},
  url = {https://openreview.net/forum?id=dJd6QxF2Ze},
}
Figure