MUST-RAG: MUSical Text Question Answering with Retrieval Augmented Generation

Daeyong Kwon, Seungheon Doh, Juhan NamarXiv 2025 · DOI · arXiv

Introduction

While Large Language Models (LLMs) like GPT-4o demonstrate remarkable general-purpose reasoning, they frequently struggle with domain-specific knowledge, particularly in the rapidly evolving and niche world of music. These models often "hallucinate" metadata or fail to recall specific discographies because music-specific data represents only a fraction of their training corpora.

Traditional domain adaptation via fine-tuning faces significant hurdles: it is computationally expensive, results in static knowledge that cannot account for new releases, and often leads to "contextual decay"—where a model becomes so reliant on its internal weights that it ignores relevant context provided in a prompt. To address these issues, we introduce MusT-RAG, a framework that treats music knowledge as an open-book task using Retrieval-Augmented Generation (RAG).

The MusT-RAG Framework

MusT-RAG combines a specialized music database with a novel fine-tuning strategy to ensure both factual accuracy and contextual flexibility. The core architecture follows a standard RAG pipeline but is optimized for the unique linguistic characteristics of music queries.

Overview of the MusT-RAG framework, illustrating the retrieval process from MusWikiDB to generate an augmented prompt for the LLM.

1. MusWikiDB and Retrieval

Central to the system is MusWikiDB, a custom vector database containing over 629,000 passages from 31,000 music-related Wikipedia pages. Interestingly, the authors found that for music, keyword-based retrieval (BM25) outperformed dense neural embeddings. This is likely due to the high density of unique entities—such as obscure band names or album titles—which keyword search handles with greater precision than semantic embeddings.

2. RAG-Style Fine-Tuning

The "secret sauce" of MusT-RAG is its training objective. Instead of standard Question-Answer fine-tuning, the model is trained on (Context, Question, Answer) triples. This explicitly teaches the LLM how to extract and synthesize information from retrieved passages, mitigating the contextual decay typically seen in domain-adapted models.

Specialized Datasets and Benchmarks

To evaluate MusT-RAG, the researchers developed two new resources:

  • MusWikiDB: A curated corpus covering seven categories: artists, genres, instruments, history, technology, theory, and forms.
  • ArtistMus: A benchmark of 1,000 multiple-choice questions focused on artist metadata. It distinguishes between "Seen" artists (included in training) and "Unseen" artists to test the model's ability to generalize to new information.

Experimental Results

The results demonstrate that MusT-RAG significantly outperforms both general-purpose LLMs and standard fine-tuned variants. On the ArtistMus benchmark, MusT-RAG achieved a factual accuracy of 82.4%, a massive leap over the base Llama 3.1 (39.0%) and notably higher than GPT-4o (67.4%).

Performance comparison of MusT-RAG against baseline models on the ArtistMus benchmark, highlighting factual and contextual accuracy.

Key Insights

  • The Fine-Tuning Paradox: Standard fine-tuning improved factual recall for known artists by only 1% but caused a 5.5% drop in contextual reasoning. In contrast, MusT-RAG's RAG-style fine-tuning improved both metrics simultaneously.
  • Efficiency of Specialization: By using a domain-specific database (MusWikiDB) rather than a general Wikipedia corpus, the system was not only more accurate but also 10x faster in retrieval time (0.0038s vs 0.0384s).
  • Robust Generalization: MusT-RAG maintained high performance (83.2% accuracy) on "Unseen" artists, proving that the model learned the process of music information retrieval rather than just memorizing facts.

Conclusion and Future Work

MusT-RAG provides a scalable, efficient, and highly accurate solution for music-specific QA. By decoupling knowledge (stored in MusWikiDB) from reasoning (the fine-tuned LLM), the system can be updated with new music releases without requiring expensive retraining.

Future research will focus on expanding the system to include audio-text joint embeddings, allowing the model to answer questions about audio characteristics like mood or instrumentation directly from sound files, and moving beyond multiple-choice evaluations to long-form conversational music critique.

Summary updated

Abstract

Recent advancements in Large language models (LLMs) have demonstrated remarkable capabilities across diverse domains. While they exhibit strong zero-shot performance on various tasks, LLMs' effectiveness in music-related applications remains limited due to the relatively small proportion of music-specific knowledge in their training data. To address this limitation, we propose MusT-RAG, a comprehensive framework based on Retrieval Augmented Generation (RAG) to adapt general-purpose LLMs for text-only music question answering (MQA) tasks. RAG is a technique that provides external knowledge to LLMs by retrieving relevant context information when generating answers to questions. To optimize RAG for the music domain, we (1) propose MusWikiDB, a music-specialized vector database for the retrieval stage, and (2) utilizes context information during both inference and fine-tuning processes to effectively transform general-purpose LLMs into music-specific models. Our experiment demonstrates that MusT-RAG significantly outperforms traditional fine-tuning approaches in enhancing LLMs' music domain adaptation capabilities, showing consistent improvements across both in-domain and out-of-domain MQA benchmarks. Additionally, our MusWikiDB proves substantially more effective than general Wikipedia corpora, delivering superior performance and computational efficiency.
BibTeX
@misc{kwon2025must,
  author = {Daeyong Kwon and Seungheon Doh and Juhan Nam},
  title = {{MUST-RAG: MUSical Text Question Answering with Retrieval Augmented Generation}},
  year = {2025},
  eprint = {2507.23334},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2507.23334},
}
Figure