Introduction
While Large Language Models (LLMs) like GPT-4o demonstrate remarkable general-purpose reasoning, they frequently struggle with domain-specific knowledge, particularly in the rapidly evolving and niche world of music. These models often "hallucinate" metadata or fail to recall specific discographies because music-specific data represents only a fraction of their training corpora.
Traditional domain adaptation via fine-tuning faces significant hurdles: it is computationally expensive, results in static knowledge that cannot account for new releases, and often leads to "contextual decay"—where a model becomes so reliant on its internal weights that it ignores relevant context provided in a prompt. To address these issues, we introduce MusT-RAG, a framework that treats music knowledge as an open-book task using Retrieval-Augmented Generation (RAG).
The MusT-RAG Framework
MusT-RAG combines a specialized music database with a novel fine-tuning strategy to ensure both factual accuracy and contextual flexibility. The core architecture follows a standard RAG pipeline but is optimized for the unique linguistic characteristics of music queries.
1. MusWikiDB and Retrieval
Central to the system is MusWikiDB, a custom vector database containing over 629,000 passages from 31,000 music-related Wikipedia pages. Interestingly, the authors found that for music, keyword-based retrieval (BM25) outperformed dense neural embeddings. This is likely due to the high density of unique entities—such as obscure band names or album titles—which keyword search handles with greater precision than semantic embeddings.
2. RAG-Style Fine-Tuning
The "secret sauce" of MusT-RAG is its training objective. Instead of standard Question-Answer fine-tuning, the model is trained on (Context, Question, Answer) triples. This explicitly teaches the LLM how to extract and synthesize information from retrieved passages, mitigating the contextual decay typically seen in domain-adapted models.
Specialized Datasets and Benchmarks
To evaluate MusT-RAG, the researchers developed two new resources:
- MusWikiDB: A curated corpus covering seven categories: artists, genres, instruments, history, technology, theory, and forms.
- ArtistMus: A benchmark of 1,000 multiple-choice questions focused on artist metadata. It distinguishes between "Seen" artists (included in training) and "Unseen" artists to test the model's ability to generalize to new information.
Experimental Results
The results demonstrate that MusT-RAG significantly outperforms both general-purpose LLMs and standard fine-tuned variants. On the ArtistMus benchmark, MusT-RAG achieved a factual accuracy of 82.4%, a massive leap over the base Llama 3.1 (39.0%) and notably higher than GPT-4o (67.4%).
Key Insights
- The Fine-Tuning Paradox: Standard fine-tuning improved factual recall for known artists by only 1% but caused a 5.5% drop in contextual reasoning. In contrast, MusT-RAG's RAG-style fine-tuning improved both metrics simultaneously.
- Efficiency of Specialization: By using a domain-specific database (MusWikiDB) rather than a general Wikipedia corpus, the system was not only more accurate but also 10x faster in retrieval time (0.0038s vs 0.0384s).
- Robust Generalization: MusT-RAG maintained high performance (83.2% accuracy) on "Unseen" artists, proving that the model learned the process of music information retrieval rather than just memorizing facts.
Conclusion and Future Work
MusT-RAG provides a scalable, efficient, and highly accurate solution for music-specific QA. By decoupling knowledge (stored in MusWikiDB) from reasoning (the fine-tuned LLM), the system can be updated with new music releases without requiring expensive retraining.
Future research will focus on expanding the system to include audio-text joint embeddings, allowing the model to answer questions about audio characteristics like mood or instrumentation directly from sound files, and moving beyond multiple-choice evaluations to long-form conversational music critique.
Summary updated