Abstract
While Large Language Models (LLMs) are increasingly integrated into music search and generation systems, their actual semantic understanding of music remains under-explored. This study evaluates off-the-shelf Transformer-based models on their ability to navigate musical taxonomies. Results indicate that these models suffer from significant prompt sensitivity, an inability to process negation, and a reliance on specific keywords rather than musical concepts.
Introduction
The "two-tower" architecture—pairing an audio encoder with a text encoder like BERT—has become a standard in Music Information Retrieval (MIR). These systems facilitate text-to-audio generation and automated captioning. However, a critical question remains: do the underlying language models possess a robust "musical compass"? For instance, can a model distinguish that a "sitar" is semantically closer to a "guitar" than a "trumpet," or is it merely performing string matching? This research investigates three primary vulnerabilities: prompt sensitivity, the "negation problem," and the distinction between label recognition and conceptual understanding.
Methodology
The researchers implemented a triplet-based evaluation framework. Each triplet consists of an Anchor, a Positive (semantically similar), and a Negative (less similar). Similarity was defined using the hierarchical structure of the AudioSet ontology. For example, "Electric Guitar" and "Acoustic Guitar" are considered closer than "Electric Guitar" and "Flute" due to their shared parent node in the ontology.
Six Transformer-based models (including MPNet, RoBERTa, and MiniLM) were tested. The evaluation process involved:
- Generating sentence embeddings for labels using 20 distinct "musically informed" prompts (e.g., "A recording of [label]").
- Calculating the cosine similarity between the Anchor-Positive and Anchor-Negative pairs.
- Defining a "correct" classification as one where the Positive label is mathematically closer to the Anchor than the Negative label.
Datasets
The study utilized the AudioSet ontology, a hierarchical collection of over 600 audio classes, to extract:
- 13,633 Music Genre triplets
- 37,640 Music Instrument triplets
Additionally, a subset of negation triplets (e.g., Anchor: "Rock", Positive: "Pop", Negative: "Not Rock") was introduced to test the models' ability to handle exclusion logic.
Key Results and Analysis
General Performance and the Negation Failure
Standard accuracy across the models ranged from 60% to 76%. While this exceeds random chance (50%), it suggests a lack of the precision required for expert-level MIR systems. The most striking finding was the failure of models to handle negation. In tests involving "not" (e.g., "Not Rock"), performance dropped significantly below random chance, with accuracies falling between 29% and 43%.
As shown in the data, models consistently fail to map the semantic meaning of negation, often treating a negated term as highly similar to the original concept. In a practical retrieval context, this implies that a user query for "music without vocals" might erroneously return vocal-heavy results.
Prompt Brittleness
The models exhibited high sensitivity to the phrasing of prompts. Minor variations, such as changing "A recording of [label]" to "The sound of [label]," resulted in significant accuracy fluctuations. This indicates that the "musical knowledge" within these models is not robust but is instead highly dependent on the linguistic contexts encountered during pre-training.
Labels vs. Definitions
The researchers observed that while providing a text definition (e.g., describing a "Cello") alongside a label improved accuracy, providing only the definition caused performance to collapse. This suggests that models are over-reliant on specific keywords (labels) rather than the underlying musical concepts described in the definitions.
Discussion and Limitations
The study identifies several hurdles for future research:
- Ontological Constraints: Tree-based ontologies struggle to model cross-branch relationships (e.g., the link between "Guitar" and "Rock music").
- Western Bias: The AudioSet ontology and manual filtering processes are predominantly biased toward Western musical traditions.
- Model Scaling: While mpnet-base generally outperformed smaller models like MiniLM, it also exhibited the highest variance across different prompts, suggesting that larger models may be more sensitive to input phrasing.
Conclusion
Current pretrained language models lack a deep semantic understanding of music, particularly regarding negation and conceptual definitions. Future work should explore more complex lexical databases like WordNet and implement multi-task learning strategies. Specifically, training models to maximize the distance between positive and negated versions of the same tag could help resolve the negation blind spot in music retrieval systems.
Summary updated