Universal Music Representations? Evaluating Foundation Models on World Music Corpora

Charilaos Papaioannou, Emmanouil Benetos, Alexandros Potamianos

This paper, titled "Universal Music Representations? Evaluating Foundation Models on World Music Corpora," investigates whether the current "foundation model" (FM) revolution in AI has truly produced a "universal" understanding of music, or if these models are just really good at Western pop.

TL;DR

The authors evaluate five state-of-the-art audio foundation models across six diverse musical traditions, ranging from Western pop to Indian classical music. While these models achieve state-of-the-art results on most datasets, their performance significantly degrades on culturally distant traditions, revealing a persistent Western-centric bias in "universal" AI.

What Problem Does This Paper Address?

In Music Information Retrieval (MIR), foundation models (like MERT or CLAP) are trained on massive amounts of data to perform various tasks like beat tracking or genre tagging. These models are often marketed as "universal," but they are predominantly trained on Western-centric datasets.

Music is not a monolithic language; different cultures have evolved distinct scales, rhythms, and semantic meanings. This paper asks a critical question: Do these models actually generalize across diverse musical traditions, or do they fail when they encounter non-Western structures? Furthermore, the authors look at how these models perform in "low-resource" scenarios—situations where we don't have thousands of labeled examples, which is common for many world music collections.

Methods

The researchers employed three complementary methodologies to test the models:

  1. Probing (Prob.): The foundation model is kept "frozen" (its internal weights don't change), acting only as a feature extractor. A small, trainable classifier is added on top to see what the model already "knows" inherently.
  2. Supervised Fine-Tuning (SFT): The last 1–2 layers of the model are unfrozen and trained alongside the classifier. This tests the model's "adaptation potential"—how easily it can learn new cultural nuances.
  3. Multi-Label Few-Shot Learning (ML-FSL): This evaluates performance when only a handful of examples (e.g., 3 per tag) are available.

The "Secret Sauce": The authors introduced a significant optimization for LC-Protonets (a few-shot learning architecture). By identifying and collapsing redundant label combinations into "Unique LCPs," they reduced inference time by up to 100x for large tag sets without losing any accuracy.

Datasets

The study used six corpora to provide a cross-cultural benchmark:

  • Western Popular: MagnaTagATune and FMA-medium.
  • Greek Folk: The Lyra dataset.
  • Turkish Classical: The Turkish-makam corpus.
  • Indian Classical: Hindustani and Carnatic collections from the CompMusic project.

The authors standardized these datasets to roughly 200 hours each to ensure a fair comparison.

Key Insights and Ideas

  • The "Universal" Gap: There is a clear performance drop as music becomes "culturally distant" from Western traditions. For example, the CLAP-Music model’s performance (ROC-AUC) plummeted from ~80% on Western music to ~61% on Carnatic music.
  • Size Isn't Everything: Surprisingly, the smaller MERT-95M model often outperformed the larger MERT-330M. The authors suggest the larger model might have a stronger Western bias or that their specific representation extraction method didn't scale perfectly with the larger architecture.
  • Pre-loaded Knowledge: In many cases, "Probing" performed nearly as well as "Fine-Tuning." This suggests that foundation models already encode a substantial amount of musical knowledge in their frozen states; fine-tuning just the last layer is often enough to unlock it.
  • Speech Matters: Models trained on both music and speech (like CLAP-M&S) significantly outperformed those trained only on music (CLAP-M), likely because speech data helps the model learn finer acoustic textures.

Key Results

  • New State-of-the-Art (SOTA): The proposed approaches achieved SOTA results on five out of the six evaluated datasets.
  • Top Performer: Qwen2-Audio (the largest model at 8.4B total parameters) was the overall winner, achieving a mean ROC-AUC of 88.98% across all datasets.
  • Efficiency Gain: The optimized few-shot learning method reduced inference time from over 2,000ms per item to just 20ms for datasets with 60 labels.
  • Few-Shot Struggle: Despite the power of FMs, they still struggle in low-resource settings compared to a simple VGG-ish baseline trained specifically on those tags, highlighting that pre-training doesn't automatically solve the "small data" problem for world music.

Limitations and Future Work

  • Hardware Constraints: Due to RAM limitations, the authors could only fine-tune the last 1–2 layers. They suggest that using LoRA (Low-Rank Adaptation) could allow for deeper tuning in future studies.
  • Task Variety: This paper focused on "tagging." Future work should explore more specialized cultural tasks, such as raga recognition in Indian music or makam estimation in Turkish music.
  • Dataset Bias: The models are still fundamentally limited by the Western-centric nature of their pre-training data. The authors call for more diverse data in the initial training phase of future foundation models.

Summary updated

Abstract

Foundation models have revolutionized music information retrieval, but questions remain about their ability to generalize across diverse musical traditions. This paper presents a comprehensive evaluation of five state-of-the-art audio foundation models across six musical corpora spanning Western popular, Greek, Turkish, and Indian classical traditions. We employ three complementary methodologies to investigate these models' cross-cultural capabilities: probing to assess inherent representations, targeted supervised fine-tuning of 1-2 layers, and multi-label few-shot learning for low-resource scenarios. Our analysis shows varying cross-cultural generalization, with larger models typically outperforming on non-Western music, though results decline for culturally distant traditions. Notably, our approaches achieve state-of-the-art performance on five out of six evaluated datasets, demonstrating the effectiveness of foundation models for world music understanding. We also find that our targeted fine-tuning approach does not consistently outperform probing across all settings, suggesting foundation models already encode substantial musical knowledge. Our evaluation framework and benchmarking results contribute to understanding how far current models are from achieving universal music representations while establishing metrics for future progress.
BibTeX
@inproceedings{papaioannou2025universal,
  author = {Charilaos Papaioannou and Emmanouil Benetos and Alexandros Potamianos},
  title = {{Universal Music Representations? Evaluating Foundation Models on World Music Corpora}},
  booktitle = {ISMIR},
  year = {2025},
  url = {https://openreview.net/forum?id=phq1Jr1mtI},
}

Loading related content...

Figure