Universal Music Representations? Evaluating Foundation Models on World Music Corpora

Charilaos Papaioannou, Emmanouil Benetos, Alexandros Potamianos

Abstract

Foundation models have revolutionized music information retrieval, but questions remain about their ability to generalize across diverse musical traditions. This paper presents a comprehensive evaluation of five state-of-the-art audio foundation models across six musical corpora spanning Western popular, Greek, Turkish, and Indian classical traditions. We employ three complementary methodologies to investigate these models' cross-cultural capabilities: probing to assess inherent representations, targeted supervised fine-tuning of 1-2 layers, and multi-label few-shot learning for low-resource scenarios. Our analysis shows varying cross-cultural generalization, with larger models typically outperforming on non-Western music, though results decline for culturally distant traditions. Notably, our approaches achieve state-of-the-art performance on five out of six evaluated datasets, demonstrating the effectiveness of foundation models for world music understanding. We also find that our targeted fine-tuning approach does not consistently outperform probing across all settings, suggesting foundation models already encode substantial musical knowledge. Our evaluation framework and benchmarking results contribute to understanding how far current models are from achieving universal music representations while establishing metrics for future progress.

Introduction

The emergence of audio foundation models (FMs) like MERT and CLAP has transformed Music Information Retrieval (MIR). While these models are often touted as "universal," they are predominantly trained on Western-centric datasets. Music, however, is not a monolithic language; distinct cultures have evolved unique scales, rhythms, and semantic structures. This study investigates whether current FMs truly generalize across diverse musical traditions or if they exhibit significant performance degradation when encountering non-Western music. Furthermore, we explore model performance in low-resource scenarios, which are common in world music collections where labeled data is scarce.

Methodology

To rigorously evaluate cross-cultural generalization, we employed three distinct methodologies:

  1. Probing (Prob.): The foundation model remains frozen, acting as a feature extractor for a small, trainable classifier. This assesses the model's inherent knowledge.
  2. Supervised Fine-Tuning (SFT): The final 1–2 layers are unfrozen and trained with the classifier to test the model's adaptation potential.
  3. Multi-Label Few-Shot Learning (ML-FSL): Performance is evaluated using only a few examples per tag (e.g., 3-shot).
Evaluation methodologies including Probing, SFT, and ML-FSL with optimized LC-Protonets.

A key technical contribution is an optimization for LC-Protonets in the few-shot setting. By identifying and collapsing redundant label combinations into "Unique LCPs," the researchers achieved up to a 100x reduction in inference time for large tag sets without sacrificing accuracy.

Experimental Setup

We benchmarked five state-of-the-art models (VGG-ish, MERT-95M, MERT-330M, CLAP-M, CLAP-M&S, and Qwen2-Audio) across six standardized corpora, each roughly 200 hours in size:

  • Western Popular: MagnaTagATune, FMA-medium.
  • Greek Folk: Lyra.
  • Turkish Classical: Turkish-makam.
  • Indian Classical: Hindustani, Carnatic.

Results and Discussion

Our evaluation revealed several critical insights into the current state of universal music representations:

The "Universal" Gap

Performance consistently declines as music becomes "culturally distant" from Western traditions. For instance, the CLAP-Music model's ROC-AUC dropped from ~80% on Western music to ~61% on Carnatic music. This highlights a persistent Western-centric bias in current foundation models.

Model Performance and Scale

Qwen2-Audio, the largest model evaluated (8.4B parameters), emerged as the top performer with a mean ROC-AUC of 88.98%. Interestingly, model size did not always correlate with performance; MERT-95M frequently outperformed the larger MERT-330M, suggesting that larger architectures may sometimes encode stronger cultural biases or require different extraction strategies.

Detailed performance metrics (ROC-AUC and mAP) across all six musical corpora for Probing and SFT.

Pre-loaded Knowledge vs. Adaptation

In many scenarios, Probing performed nearly as well as SFT. This indicates that foundation models already encode substantial musical knowledge in their frozen states. Fine-tuning just the final layers is often sufficient to unlock this knowledge for specific cultural contexts. Additionally, models trained on both music and speech (CLAP-M&S) outperformed music-only models (CLAP-M), likely due to the richer acoustic textures learned from speech data.

Conclusion

Our work establishes a new state-of-the-art on five of the six evaluated datasets. However, the struggle of FMs in few-shot settings compared to simple baselines suggests that pre-training alone does not solve the "small data" problem for world music. Future research should leverage techniques like LoRA for deeper fine-tuning and expand evaluations to specialized cultural tasks like raga or makam recognition. Ultimately, achieving true universal music representation will require more diverse and inclusive pre-training datasets.

Summary updated

Abstract

Foundation models have revolutionized music information retrieval, but questions remain about their ability to generalize across diverse musical traditions. This paper presents a comprehensive evaluation of five state-of-the-art audio foundation models across six musical corpora spanning Western popular, Greek, Turkish, and Indian classical traditions. We employ three complementary methodologies to investigate these models' cross-cultural capabilities: probing to assess inherent representations, targeted supervised fine-tuning of 1-2 layers, and multi-label few-shot learning for low-resource scenarios. Our analysis shows varying cross-cultural generalization, with larger models typically outperforming on non-Western music, though results decline for culturally distant traditions. Notably, our approaches achieve state-of-the-art performance on five out of six evaluated datasets, demonstrating the effectiveness of foundation models for world music understanding. We also find that our targeted fine-tuning approach does not consistently outperform probing across all settings, suggesting foundation models already encode substantial musical knowledge. Our evaluation framework and benchmarking results contribute to understanding how far current models are from achieving universal music representations while establishing metrics for future progress.
BibTeX
@inproceedings{papaioannou2025universal,
  author = {Charilaos Papaioannou and Emmanouil Benetos and Alexandros Potamianos},
  title = {{Universal Music Representations? Evaluating Foundation Models on World Music Corpora}},
  booktitle = {ISMIR},
  year = {2025},
  url = {https://openreview.net/forum?id=phq1Jr1mtI},
}
Figure