Abstract
MuseChat is a novel conversational system designed to provide personalized music recommendations for video content through natural language dialogue. Traditional recommendation systems often operate as "black boxes," offering limited user control and lacking transparency. MuseChat addresses these issues by integrating a tri-modal retrieval architecture (MVT-Fusion) with a Large Language Model (LLM) to enable iterative refinement of suggestions and human-like reasoning. Experimental results on a newly curated large-scale dataset demonstrate that MuseChat significantly outperforms existing methods in both retrieval accuracy and user satisfaction.
Introduction
Most existing music recommendation systems for video content are "one-shot" and "black-box." They analyze a video and generate a list of songs based on content compatibility, often ignoring the specific, nuanced preferences of the user. If a user is dissatisfied with the initial suggestions, there is typically no mechanism to provide feedback or explain why a specific track was chosen. Furthermore, these systems frequently struggle with the "cold-start" problem for new users.
MuseChat addresses these gaps by introducing interactivity into the recommendation process. Users can upload a video, receive a suggestion, and then provide natural language feedback (e.g., "I want something more upbeat with female vocals") to steer the system toward the perfect track. This conversational approach allows for iterative refinement and provides human-like reasoning for its choices.
System Architecture
The MuseChat architecture consists of two primary components: the Music Recommendation Module (MVT-Fusion) and the Sentence Generator Module.
Music Recommendation Module (MVT-Fusion)
This tri-modal architecture is designed to align video, music, and text in a shared embedding space. The module utilizes a pretrained Audio Spectrogram Transformer (AST) for music encoding, CLIP for video frames, and a frozen CLIP text encoder for user prompts.
The system employs a "late fusion" strategy. It uses self-attention layers to capture intra-modal dynamics and cross-modal attention (Audio-to-Text and Text-to-Audio) to fuse the candidate music and user feedback before integrating visual features. The model is trained using a contrastive multiview coding loss (a variant of InfoNCE), which pushes the "goal" embedding (video + feedback + previous music) closer to the target music track in the embedding space.
Sentence Generator Module
To provide reasoning for its recommendations, the authors developed a multi-modal LLM based on Vicuna-7B. Averaged music features from the AST encoder are projected into the LLM’s text embedding space using a linear layer. The model is then fine-tuned using Low-Rank Adaptation (LoRA) to generate justifications based on the music's attributes (mood, genre, instruments) and the video context.
Datasets
The authors introduced a massive new dataset specifically for this task: the Conversational Music Recommendation Dataset. Derived from YouTube8M, it contains 98,206 quartets consisting of a video, original music, a "candidate" (initial) music suggestion, and a two-turn simulated conversation. To simulate realistic user feedback, music tags from the MagnaTagATune (MTT) and Million Song Dataset (MSD) were processed by GPT-3.5 to generate natural-sounding dialogues that bridge the gap between a "wrong" initial suggestion and the "correct" target track.
Key Insights and Results
Experimental results highlight several critical findings:
- The Power of Feedback: User feedback significantly improves retrieval. While the system is effective at picking music from video alone, adding a single turn of natural language refinement causes a massive jump in accuracy.
- Modality Separation: Keeping visual information as an independent branch in the fusion process, rather than forcing it through immediate cross-attention with text/audio, was found to be crucial for performance. Video provides a foundational context that should not be muddled by specific textual constraints too early.
- Transparency through Reasoning: By using an LLM to explain why a song was picked, the system improves user trust. Human evaluations showed that MuseChat significantly improves satisfaction compared to black-box baselines.
Quantitative Performance
MuseChat demonstrated superior performance across retrieval metrics:
- Retrieval Accuracy: MuseChat achieved a Recall@1 (R@1) of 32.79%, significantly higher than the "Cross-Attn Fusion" baseline (25.71%).
- Success Rate: The system reached a 40.49% success rate in finding the target music within two turns.
- Human Evaluation: In a 5-point Mean Opinion Score (MOS) test, MuseChat scored 4.63 for correctness and 4.22 for musicality.
Conclusion
MuseChat represents a significant step toward interactive, transparent music recommendation. While currently focused on two-turn interactions, the architecture is extensible to longer dialogues. Future work will explore deeper integration between the recommendation and reasoning modules, potentially using a single multi-modal LLM to handle both retrieval and generation, as well as incorporating long-term user history for further personalization.
Summary updated