MuseChat: A Conversational Music Recommendation System for Videos

Zhikang Dong, Bin Chen, Xiulong Liu, Pawel Polak, Peng Zhang

Abstract

MuseChat is a novel conversational system designed to provide personalized music recommendations for video content through natural language dialogue. Traditional recommendation systems often operate as "black boxes," offering limited user control and lacking transparency. MuseChat addresses these issues by integrating a tri-modal retrieval architecture (MVT-Fusion) with a Large Language Model (LLM) to enable iterative refinement of suggestions and human-like reasoning. Experimental results on a newly curated large-scale dataset demonstrate that MuseChat significantly outperforms existing methods in both retrieval accuracy and user satisfaction.

Introduction

Most existing music recommendation systems for video content are "one-shot" and "black-box." They analyze a video and generate a list of songs based on content compatibility, often ignoring the specific, nuanced preferences of the user. If a user is dissatisfied with the initial suggestions, there is typically no mechanism to provide feedback or explain why a specific track was chosen. Furthermore, these systems frequently struggle with the "cold-start" problem for new users.

MuseChat addresses these gaps by introducing interactivity into the recommendation process. Users can upload a video, receive a suggestion, and then provide natural language feedback (e.g., "I want something more upbeat with female vocals") to steer the system toward the perfect track. This conversational approach allows for iterative refinement and provides human-like reasoning for its choices.

Overview of the MuseChat conversational recommendation loop and system architecture.

System Architecture

The MuseChat architecture consists of two primary components: the Music Recommendation Module (MVT-Fusion) and the Sentence Generator Module.

Music Recommendation Module (MVT-Fusion)

This tri-modal architecture is designed to align video, music, and text in a shared embedding space. The module utilizes a pretrained Audio Spectrogram Transformer (AST) for music encoding, CLIP for video frames, and a frozen CLIP text encoder for user prompts.

The system employs a "late fusion" strategy. It uses self-attention layers to capture intra-modal dynamics and cross-modal attention (Audio-to-Text and Text-to-Audio) to fuse the candidate music and user feedback before integrating visual features. The model is trained using a contrastive multiview coding loss (a variant of InfoNCE), which pushes the "goal" embedding (video + feedback + previous music) closer to the target music track in the embedding space.

The MVT-Fusion architecture (left) and the detailed fusion module (right) for tri-modal music retrieval.

Sentence Generator Module

To provide reasoning for its recommendations, the authors developed a multi-modal LLM based on Vicuna-7B. Averaged music features from the AST encoder are projected into the LLM’s text embedding space using a linear layer. The model is then fine-tuned using Low-Rank Adaptation (LoRA) to generate justifications based on the music's attributes (mood, genre, instruments) and the video context.

Datasets

The authors introduced a massive new dataset specifically for this task: the Conversational Music Recommendation Dataset. Derived from YouTube8M, it contains 98,206 quartets consisting of a video, original music, a "candidate" (initial) music suggestion, and a two-turn simulated conversation. To simulate realistic user feedback, music tags from the MagnaTagATune (MTT) and Million Song Dataset (MSD) were processed by GPT-3.5 to generate natural-sounding dialogues that bridge the gap between a "wrong" initial suggestion and the "correct" target track.

Key Insights and Results

Experimental results highlight several critical findings:

  • The Power of Feedback: User feedback significantly improves retrieval. While the system is effective at picking music from video alone, adding a single turn of natural language refinement causes a massive jump in accuracy.
  • Modality Separation: Keeping visual information as an independent branch in the fusion process, rather than forcing it through immediate cross-attention with text/audio, was found to be crucial for performance. Video provides a foundational context that should not be muddled by specific textual constraints too early.
  • Transparency through Reasoning: By using an LLM to explain why a song was picked, the system improves user trust. Human evaluations showed that MuseChat significantly improves satisfaction compared to black-box baselines.

Quantitative Performance

MuseChat demonstrated superior performance across retrieval metrics:

  • Retrieval Accuracy: MuseChat achieved a Recall@1 (R@1) of 32.79%, significantly higher than the "Cross-Attn Fusion" baseline (25.71%).
  • Success Rate: The system reached a 40.49% success rate in finding the target music within two turns.
  • Human Evaluation: In a 5-point Mean Opinion Score (MOS) test, MuseChat scored 4.63 for correctness and 4.22 for musicality.

Conclusion

MuseChat represents a significant step toward interactive, transparent music recommendation. While currently focused on two-turn interactions, the architecture is extensible to longer dialogues. Future work will explore deeper integration between the recommendation and reasoning modules, potentially using a single multi-modal LLM to handle both retrieval and generation, as well as incorporating long-term user history for further personalization.

Summary updated

Abstract

We introduce MuseChat, an innovative dialog-based music recommendation system. This unique platform not only offers interactive user engagement but also suggests music tailored for input videos, so that users can refine and personalize their music selections. In contrast, previous systems predominantly emphasized content compatibility, often overlooking the nuances of users' individual preferences. For example, all the datasets only provide basic music-video pairings or such pairings with textual music descriptions. To address this gap, our research offers three contributions. First, we devise a conversation-synthesis method that simulates a two-turn interaction between a user and a recommendation system, which leverages pre-trained music tags and artist information. In this interaction, users submit a video to the system, which then suggests a suitable music piece with a rationale. Afterwards, users communicate their musical preferences, and the system presents a refined music recommendation with reasoning. Second, we introduce a multi-modal recommendation engine that matches music either by aligning it with visual cues from the video or by harmonizing visual information, feedback from previously recommended music, and the user's textual input. Third, we bridge music representations and textual data with a Large Language Model(Vicuna-7B). This alignment equips MuseChat to deliver music recommendations and their underlying reasoning in a manner resembling human communication. Our evaluations show that MuseChat surpasses existing state-of-the-art models in music retrieval tasks and pioneers the integration of the recommendation process within a natural language framework.
BibTeX
@inproceedings{dong2023musechat,
  author = {Zhikang Dong and Bin Chen and Xiulong Liu and Pawel Polak and Peng Zhang},
  title = {{MuseChat: A Conversational Music Recommendation System for Videos}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2023},
  doi = {10.1109/cvpr52733.2024.01214},
}
Figure