Introduction
Music Information Retrieval (MIR) has long been characterized by a fragmented landscape. Traditional systems are often specialized for a single modality—such as text-to-audio or symbolic music search—and rarely facilitate cross-modal interaction. Furthermore, the field is heavily biased toward English-centric datasets, neglecting the rich, multilingual diversity of global musical culture. These "silos" are primarily the result of three challenges: the structural differences between sheet music, MIDI, and raw audio; a severe scarcity of high-quality paired data; and a lack of descriptive metadata in non-English languages.
CLaMP 3 addresses these issues by introducing a universal framework that aligns sheet music, performance signals (MIDI), and audio recordings into a single, unified representation space. By using multilingual text as a semantic "bridge," the model achieves state-of-the-art performance and demonstrates emergent alignment—the ability to link modalities like sheet music and audio even without direct paired training data.
The CLaMP 3 Architecture
The core innovation of CLaMP 3 is its multi-stage contrastive learning strategy. Rather than attempting to align all modalities simultaneously, which often leads to training instability, the researchers utilized text as a stable semantic anchor. The architecture employs three specialized encoders:
- Multilingual Text: XLM-RoBERTa (XLM-R) provides a robust foundation for cross-lingual generalization.
- Symbolic Music: The M3 encoder processes music in ABC notation and MIDI formats.
- Audio Music: A combination of the MERT model and a Transformer architecture extracts features from raw audio.
Training is executed in four distinct stages to ensure balanced alignment. First, the symbolic music encoder is aligned with the text encoder. Next, the text encoder is frozen while the audio encoder is aligned to it. Finally, iterative refinement stages are performed to prevent the model from "forgetting" one modality while learning another. This approach allows the model to perform zero-shot cross-modal retrieval; for example, a user can use a sheet music snippet to find a corresponding audio recording despite the model never seeing a direct sheet-to-audio pair during training.
Dataset Innovation: M4-RAG and WikiMT-X
To power this framework, the researchers developed two major data resources. The first is M4-RAG (Million-scale Multilingual Music Metadata), a web-scale dataset containing 2.31 million music-text pairs across 27 languages. To overcome the lack of rich descriptions in existing metadata, the team employed Retrieval-Augmented Generation (RAG). By searching the web for basic metadata (like titles and artists) and processing the results with a Large Language Model (Qwen2.5-72B), they synthesized high-quality, long-form descriptions that provide a much richer training signal than simple genre tags.
Complementing this is WikiMT-X, a new "gold standard" benchmark for evaluation. It consists of 1,000 triplets—each containing sheet music, audio, and detailed text annotations—covering musical analysis, historical background, and visual scene descriptions.
Experimental Results and Key Insights
CLaMP 3 sets a new benchmark for MIR performance across several metrics:
- State-of-the-Art Retrieval: The model achieved a Mean Reciprocal Rank (MRR) of 0.4498 on symbolic music retrieval (WikiMT) and 0.1985 on audio retrieval (SDD), significantly outperforming previous models such as CLAP and TTMR++.
- Cross-Lingual Mastery: Leveraging the pre-trained knowledge of XLM-R, CLaMP 3 demonstrated impressive performance in languages not present in its training set, such as Amharic and Finnish.
- Emergent Alignment: The model successfully performed Sheet Music $\rightarrow$ Audio retrieval with an MRR of 0.0578, proving that a central text hub can effectively link unaligned modalities.
Visualizations using t-SNE suggest that the model organizes different modalities in a mirrored fashion around the central text cluster, confirming that the text acts as a semantic anchor that preserves musical meaning across formats.
Conclusion and Future Directions
While CLaMP 3 represents a significant leap toward universal MIR, some limitations remain. The model currently uses global embeddings, which can lead to "temporal blindness" regarding fine-grained motifs within a track. Additionally, the link between audio and symbolic music, while functional, remains weaker than the links involving text. Future research will likely focus on capturing temporal dynamics and incorporating more direct audio-symbolic pairs to further tighten the alignment between all musical modalities.
Summary updated