TL;DR
Researchers at Dolby Laboratories have developed Control-MVR, a framework that uses semi-supervised contrastive learning to match music to video. By blending self-supervised "artistic correspondence" with supervised genre labels, the system allows users to use a single "control knob" ($\alpha$) at inference time to prioritize either exact audiovisual alignment or broad thematic genre matching.
What Problem Does This Paper Address?
Content creators often struggle to find the "perfect" track for a video—a process that is traditionally manual and time-consuming. While AI has made strides in cross-modal retrieval, existing models usually fall into two camps:
- Self-supervised models: These learn from the natural pairing of audio and video in raw data. They are great at capturing general "vibes" but often lack the domain-specific precision of human-defined categories like genre or mood.
- Supervised models: These are trained on explicit labels (e.g., "this is a Jazz video"). While precise, they can overfit to labels and ignore the nuanced, non-verbal artistic relationships between visuals and sound.
The authors address the lack of a system that combines these two strengths while giving the user explicit control over which one to prioritize during the search process.
Methods
The authors introduce Control-MVR (Controllable Music-Video Retrieval), which utilizes a dual-branch architecture to process audio and video separately.
- Feature Extraction: The model uses high-quality, frozen "backbones": MERT for audio (a large-scale self-supervised music model) and CLIP (ViT-B/32) for video frames.
- The Dual-Path Architecture: Within each branch, the model splits into two task-specific paths:
- Self-Supervised Path ($q_{ssl}$): Uses an InfoNCE loss to "pull together" specific audio-video pairs from the same source while pushing others away.
- Supervised Path ($q_{sup}$): Uses a SupCon (Supervised Contrastive) loss to group all clips belonging to the same genre (e.g., all "Rock" videos) together in the embedding space.
- The Control Knob ($\alpha$): The final output embedding is a linear combination of these two paths: $z = (1-\alpha)p_{ssl} + \alpha p_{sup}$.
- During training, $\alpha$ is a hyperparameter.
- At inference, the user can slide $\alpha$ from 0 to 1. At 0, the system focuses purely on self-supervised audiovisual patterns; at 1, it focuses purely on supervised genre knowledge.
Datasets
The study uses a curated subset of AudioSet, consisting of 106,400 music videos (10-second clips).
- Genre Taxonomy: Because AudioSet's original labels are messy, the authors condensed them into 11 broad categories (e.g., Country, Classical, Hip-Hop, R&B, etc.) to provide a cleaner training signal for the supervised component.
- Split: 87,710 videos for training, 10,000 for validation, and 8,000 for testing.
Key Insights and Ideas
- Synergy, Not Competition: One of the most important findings is that supervised information actually improves self-supervised retrieval. The "Optimal $\alpha$" for self-supervised tasks wasn't 0.0, but rather around 0.4. This suggests that knowing the genre helps the model better understand the underlying artistic correspondence.
- Dynamic Controllability: Most retrieval models are "static"—once trained, their behavior is fixed. Control-MVR’s ability to adjust the retrieval logic at runtime without retraining is a significant UX advantage for creative tools.
- Architecture Matters: The authors found that a simple semi-supervised baseline (just adding the losses together) didn't perform as well as their specific dual-path architecture. This indicates that explicitly separating the "content" (self-supervised) from the "category" (supervised) before combining them is crucial.
Key Results
Control-MVR achieved State-of-the-Art (SOTA) or near-SOTA performance across multiple benchmarks:
- Self-Supervised Retrieval: With an optimal $\alpha$ of 0.4, the model outperformed established baselines like Wav2CLIP and MVPt, achieving a Recall@10 of 10.42% for Video $\to$ Music retrieval.
- Genre-Supervised Retrieval: With an $\alpha$ of 0.8, the model achieved a Precision@10 of 43.24%, significantly outperforming purely self-supervised methods (which hovered around 33%).
- Robustness: The supervised baseline tended to overfit to genre labels, performing poorly on self-supervised tasks. Control-MVR remained competitive across both types of evaluation.
Limitations and Future Work
- Label Diversity: Currently, the model only uses genre labels. The authors acknowledge that music is defined by more than just genre, such as emotion, instrumentation, and tempo.
- Frozen Encoders: The audio and video backbones (MERT and CLIP) are kept frozen. Fine-tuning these encoders specifically for the music-video domain could yield even higher performance.
- Language Integration: The authors plan to explore adding language-based queries (e.g., "find a happy song for this sunny beach video") into the framework to allow for even more granular control.
Summary updated