Semi-Supervised Contrastive Learning for Controllable Video-to-Music Retrieval

Shanti Stewart, Gouthaman Kv, Lie Lu, Andrea Fanelli

TL;DR

Researchers at Dolby Laboratories have developed Control-MVR, a framework that uses semi-supervised contrastive learning to match music to video. By blending self-supervised "artistic correspondence" with supervised genre labels, the system allows users to use a single "control knob" ($\alpha$) at inference time to prioritize either exact audiovisual alignment or broad thematic genre matching.

What Problem Does This Paper Address?

Content creators often struggle to find the "perfect" track for a video—a process that is traditionally manual and time-consuming. While AI has made strides in cross-modal retrieval, existing models usually fall into two camps:

  1. Self-supervised models: These learn from the natural pairing of audio and video in raw data. They are great at capturing general "vibes" but often lack the domain-specific precision of human-defined categories like genre or mood.
  2. Supervised models: These are trained on explicit labels (e.g., "this is a Jazz video"). While precise, they can overfit to labels and ignore the nuanced, non-verbal artistic relationships between visuals and sound.

The authors address the lack of a system that combines these two strengths while giving the user explicit control over which one to prioritize during the search process.

Methods

The authors introduce Control-MVR (Controllable Music-Video Retrieval), which utilizes a dual-branch architecture to process audio and video separately.

  • Feature Extraction: The model uses high-quality, frozen "backbones": MERT for audio (a large-scale self-supervised music model) and CLIP (ViT-B/32) for video frames.
  • The Dual-Path Architecture: Within each branch, the model splits into two task-specific paths:
  • Self-Supervised Path ($q_{ssl}$): Uses an InfoNCE loss to "pull together" specific audio-video pairs from the same source while pushing others away.
  • Supervised Path ($q_{sup}$): Uses a SupCon (Supervised Contrastive) loss to group all clips belonging to the same genre (e.g., all "Rock" videos) together in the embedding space.
  • The Control Knob ($\alpha$): The final output embedding is a linear combination of these two paths: $z = (1-\alpha)p_{ssl} + \alpha p_{sup}$.
  • During training, $\alpha$ is a hyperparameter.
  • At inference, the user can slide $\alpha$ from 0 to 1. At 0, the system focuses purely on self-supervised audiovisual patterns; at 1, it focuses purely on supervised genre knowledge.

Datasets

The study uses a curated subset of AudioSet, consisting of 106,400 music videos (10-second clips).

  • Genre Taxonomy: Because AudioSet's original labels are messy, the authors condensed them into 11 broad categories (e.g., Country, Classical, Hip-Hop, R&B, etc.) to provide a cleaner training signal for the supervised component.
  • Split: 87,710 videos for training, 10,000 for validation, and 8,000 for testing.

Key Insights and Ideas

  • Synergy, Not Competition: One of the most important findings is that supervised information actually improves self-supervised retrieval. The "Optimal $\alpha$" for self-supervised tasks wasn't 0.0, but rather around 0.4. This suggests that knowing the genre helps the model better understand the underlying artistic correspondence.
  • Dynamic Controllability: Most retrieval models are "static"—once trained, their behavior is fixed. Control-MVR’s ability to adjust the retrieval logic at runtime without retraining is a significant UX advantage for creative tools.
  • Architecture Matters: The authors found that a simple semi-supervised baseline (just adding the losses together) didn't perform as well as their specific dual-path architecture. This indicates that explicitly separating the "content" (self-supervised) from the "category" (supervised) before combining them is crucial.

Key Results

Control-MVR achieved State-of-the-Art (SOTA) or near-SOTA performance across multiple benchmarks:

  • Self-Supervised Retrieval: With an optimal $\alpha$ of 0.4, the model outperformed established baselines like Wav2CLIP and MVPt, achieving a Recall@10 of 10.42% for Video $\to$ Music retrieval.
  • Genre-Supervised Retrieval: With an $\alpha$ of 0.8, the model achieved a Precision@10 of 43.24%, significantly outperforming purely self-supervised methods (which hovered around 33%).
  • Robustness: The supervised baseline tended to overfit to genre labels, performing poorly on self-supervised tasks. Control-MVR remained competitive across both types of evaluation.

Limitations and Future Work

  • Label Diversity: Currently, the model only uses genre labels. The authors acknowledge that music is defined by more than just genre, such as emotion, instrumentation, and tempo.
  • Frozen Encoders: The audio and video backbones (MERT and CLIP) are kept frozen. Fine-tuning these encoders specifically for the music-video domain could yield even higher performance.
  • Language Integration: The authors plan to explore adding language-based queries (e.g., "find a happy song for this sunny beach video") into the framework to allow for even more granular control.

Summary updated

Abstract

Content creators often use music to enhance their videos, from soundtracks in movies to background music in video blogs and social media content. However, identifying the best music for a video can be a difficult and time-consuming task. To address this challenge, we propose a novel framework for automatically retrieving a matching music clip for a given video, and vice versa. Our approach leverages annotated music labels, as well as the inherent artistic correspondence between visual and music elements. Distinct from previous cross-modal music retrieval works, our method combines both self-supervised and supervised training objectives. We use self-supervised and label-supervised contrastive learning to train a joint embedding space between music and video. We show the effectiveness of our approach by using music genre labels for the supervised training component, and our framework can be generalized to other music annotations (e.g., emotion, instrument, etc.). Furthermore, our method enables fine-grained control over how much the retrieval process focuses on self-supervised vs. label information at inference time. We evaluate the learned embeddings through a variety of video-to-music and music-to-video retrieval tasks. Our experiments show that the proposed approach successfully combines self-supervised and supervised objectives and is effective for controllable music-video retrieval.
BibTeX
@inproceedings{stewart2025semi,
  author = {Shanti Stewart and Gouthaman KV and Lie Lu and Andrea Fanelli},
  title = {{Semi-Supervised Contrastive Learning for Controllable Video-to-Music Retrieval}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2025},
  doi = {10.1109/icassp49660.2025.10890382},
}

Loading related content...

Figure