Annotator Subjectivity in the MusicCaps Dataset

Minhee Lee, Seungheon Doh, Dasaem JeongHcmir@ismir 2023

Introduction

As multi-modal AI research expands from images to music, high-quality datasets pairing audio with descriptive text have become essential. However, unlike objective tags (e.g., "Jazz," "120 BPM"), free-form captions allow for creative expression, which introduces significant annotator subjectivity. If one annotator focuses on emotional "mood" while another prioritizes "technical instrumentation," a model trained on this data might inadvertently learn individual stylistic quirks rather than the underlying musical content. This paper investigates the presence of such subjectivity in the popular MusicCaps dataset and its impact on training audio-text joint embedding spaces.

Methodology

The researchers analyzed the MusicCaps dataset, which consists of 5.5k music-text pairs annotated by ten professional musicians. They employed a multi-stage approach to detect and quantify subjectivity:

  1. Semantic Embedding Visualization: Captions were embedded using a pre-trained BERT model, and audio clips using VGGish. These were projected into 2D space using UMAP to detect author-based clustering.
  2. Annotator Prediction: A Random Forest classifier was trained to predict the annotator based solely on text or audio embeddings.
  3. Cross-Modal Retrieval: Dual-encoder models were trained on both the full dataset (baseline) and annotator-specific subsets to measure how stylistic alignment affects retrieval accuracy (Recall@10).

The "Annotator Signature"

The most striking evidence of subjectivity appeared in the embedding space. While audio embeddings appeared as a disorganized cloud—confirming that audio clips were randomly assigned to annotators—the text embeddings formed distinct clusters based on the author.

UMAP projections showing that text embeddings (right) cluster by annotator, while audio embeddings (left) do not.

This "annotator signature" suggests that the way a person writes is often more mathematically distinct than the musical content they are describing. This is further supported by keyword analysis: for instance, one annotator included "theme" descriptions (e.g., "this could be used in a movie scene...") in nearly 95% of their captions, while another never used them. A Random Forest classifier achieved an F1 score of 0.76 when predicting the annotator from text, compared to just 0.08 from audio.

Impact on Retrieval Performance

Subjectivity creates an asymmetric bias in cross-modal retrieval tasks. When models are trained on a specific annotator's style, they tend to overfit to that individual's linguistic patterns.

Heatmaps showing retrieval performance (R@10) for audio-to-text (left) and text-to-audio (right) tasks across different annotator-specific models.

In audio-to-text retrieval, models trained on a specific person's data performed exceptionally well for that person (the diagonal in the left heatmap) because the learned "style" helps narrow the search space. However, these models generalized poorly to other annotators. Conversely, in text-to-audio retrieval, this stylistic bias offered no benefit, as the query's style does not provide additional information about the target audio. The baseline model, trained on all annotators, generally proved superior for general text-to-audio tasks.

Conclusion and Future Work

The study demonstrates that MusicCaps is heavily influenced by the subjective styles of its contributors. While a diverse "mixture of styles" in the baseline model generally helps text-to-audio tasks, the presence of these signatures can lead to misleading performance metrics and poor generalization. Future work will focus on developing "de-biasing" strategies to ensure that multi-modal models learn the true relationship between sound and language, rather than just the habits of the people describing them.

Summary updated

BibTeX
@inproceedings{lee2023annotator,
  author = {Minhee Lee and Seungheon Doh and Dasaem Jeong},
  title = {{Annotator Subjectivity in the MusicCaps Dataset}},
  booktitle = {Hcmir@ismir},
  year = {2023},
}
Figure