Semi-Supervised Contrastive Learning for Controllable Video-to-Music Retrieval

Shanti Stewart, Gouthaman Kv, Lie Lu, Andrea Fanelli

Abstract

Content creators often use music to enhance their videos, from soundtracks in movies to background music in video blogs and social media content. However, identifying the best music for a video can be a difficult and time-consuming task. To address this challenge, we propose a novel framework for automatically retrieving a matching music clip for a given video, and vice versa. Our approach leverages annotated music labels, as well as the inherent artistic correspondence between visual and music elements. Distinct from previous cross-modal music retrieval works, our method combines both self-supervised and supervised training objectives. We use self-supervised and label-supervised contrastive learning to train a joint embedding space between music and video. We show the effectiveness of our approach by using music genre labels for the supervised training component, and our framework can be generalized to other music annotations (e.g., emotion, instrument, etc.). Furthermore, our method enables fine-grained control over how much the retrieval process focuses on self-supervised vs. label information at inference time. We evaluate the learned embeddings through a variety of video-to-music and music-to-video retrieval tasks. Our experiments show that the proposed approach successfully combines self-supervised and supervised objectives and is effective for controllable music-video retrieval.

Introduction

Finding the "perfect" track for a video is a traditionally manual and time-consuming process for content creators. While AI-driven cross-modal retrieval has advanced, existing models typically fall into two categories: self-supervised models that capture general "vibes" from raw audiovisual pairings but lack domain-specific precision, and supervised models that are precise regarding labels (e.g., genre) but often ignore nuanced artistic relationships.

To bridge this gap, researchers at Dolby Laboratories have developed Control-MVR (Controllable Music-Video Retrieval). This framework utilizes semi-supervised contrastive learning to blend self-supervised artistic correspondence with supervised genre labels. A key innovation is the inclusion of a "control knob" ($\alpha$) that allows users at inference time to prioritize either exact audiovisual alignment or broad thematic genre matching.

Methodology

Control-MVR employs a dual-branch architecture to process audio and video modalities independently.

Feature Extraction

The model leverages high-quality, frozen backbones: MERT for audio (a large-scale self-supervised music model) and CLIP (ViT-B/32) for video frames.

Dual-Path Architecture

Within each modality branch, the model splits into two task-specific paths:

  1. Self-Supervised Path ($q_{ssl}$): Employs an InfoNCE loss to align specific audio-video pairs from the same source.
  2. Supervised Path ($q_{sup}$): Uses a Supervised Contrastive (SupCon) loss to group clips belonging to the same genre in the embedding space.
The Control-MVR architecture showing the dual-path processing for audio and video branches.

Controllable Inference

The final output embedding $z$ is a linear combination of these paths: $z = (1-\alpha)p_{ssl} + \alpha p_{sup}$. While $\alpha$ is a hyperparameter during training, it serves as a user-adjustable slider at inference. Setting $\alpha=0$ focuses purely on self-supervised audiovisual patterns, while $\alpha=1$ prioritizes supervised genre knowledge.

Experimental Setup

We evaluated our framework using a curated subset of AudioSet, comprising 106,400 music video clips (10 seconds each). To provide a clean training signal, AudioSet's original labels were condensed into 11 broad genre categories, such as Country, Classical, Hip-Hop, and R&B. The dataset was split into 87,710 training, 10,000 validation, and 8,000 test samples.

Results and Discussion

Control-MVR achieves state-of-the-art (SOTA) or near-SOTA performance across multiple retrieval benchmarks.

Performance Metrics

As shown in the results table, Control-MVR with an optimal $\alpha$ of 0.4 achieved a Recall@10 of 10.42% for Video $\to$ Music retrieval, outperforming established baselines like Wav2CLIP and MVPt. For genre-supervised tasks, an $\alpha$ of 0.8 yielded a Precision@10 of 43.24%, significantly higher than purely self-supervised methods.

Comparison of retrieval performance across different training methods and alpha settings.

Key Insights

  • Synergy of Objectives: Supervised information improves self-supervised retrieval. The optimal $\alpha$ for self-supervised tasks was approximately 0.4 rather than 0.0, suggesting that genre knowledge helps the model better understand underlying artistic correspondences.
  • Dynamic Controllability: Unlike static models, Control-MVR allows runtime adjustment of retrieval logic without retraining, offering a significant user experience advantage.
  • Architectural Separation: Explicitly separating "content" (SSL) from "category" (Supervised) before combination proved superior to simple multi-loss baselines.

Conclusion and Future Work

Control-MVR demonstrates the effectiveness of semi-supervised contrastive learning for controllable music-video retrieval. Future work will explore expanding the label space to include emotion, instrumentation, and tempo. Additionally, we plan to investigate fine-tuning the frozen encoders and integrating natural language queries to provide even more granular control for creators.

Summary updated

Abstract

Content creators often use music to enhance their videos, from soundtracks in movies to background music in video blogs and social media content. However, identifying the best music for a video can be a difficult and time-consuming task. To address this challenge, we propose a novel framework for automatically retrieving a matching music clip for a given video, and vice versa. Our approach leverages annotated music labels, as well as the inherent artistic correspondence between visual and music elements. Distinct from previous cross-modal music retrieval works, our method combines both self-supervised and supervised training objectives. We use self-supervised and label-supervised contrastive learning to train a joint embedding space between music and video. We show the effectiveness of our approach by using music genre labels for the supervised training component, and our framework can be generalized to other music annotations (e.g., emotion, instrument, etc.). Furthermore, our method enables fine-grained control over how much the retrieval process focuses on self-supervised vs. label information at inference time. We evaluate the learned embeddings through a variety of video-to-music and music-to-video retrieval tasks. Our experiments show that the proposed approach successfully combines self-supervised and supervised objectives and is effective for controllable music-video retrieval.
BibTeX
@inproceedings{stewart2025semi,
  author = {Shanti Stewart and Gouthaman KV and Lie Lu and Andrea Fanelli},
  title = {{Semi-Supervised Contrastive Learning for Controllable Video-to-Music Retrieval}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2025},
  doi = {10.1109/icassp49660.2025.10890382},
}
Figure