Abstract
With the rise of short videos, the demand for selecting appropriate background music (BGM) for a video has increased significantly, and the video-music retrieval (VMR) task has gradually drawn much attention from the research community. Like other cross-modal learning tasks, existing VMR approaches usually attempt to measure the similarity between the video and music in the feature space. However, they (1) neglect the inevitable label noise and (2) neglect to enhance the ability to capture critical video clips. In this paper, we propose a novel saliency-based self-training framework, termed SSVMR. Specifically, we explore how to fully make use of the information contained in the training dataset by applying a semi-supervised method to suppress the adverse impact of label noise, where a self-training approach is adopted. In addition, we propose to capture the saliency of the video by mixing two videos at the span level and preserving the locality of the two original videos. Inspired by back-translation in NLP, we also conduct back retrieval to obtain more training data. Experimental results on the MVD dataset show that our SSVMR achieves state-of-the-art performance by a large margin, obtaining a relative improvement of 34.8% over the previous best model in terms of R@1.
Introduction
The explosion of short-form video content on platforms like TikTok and Instagram Reels has made background music (BGM) selection a critical task for creators. Automating Video-Music Retrieval (VMR) is a complex cross-modal challenge because the relationship between visual and auditory signals is often subjective and non-linear.
Existing models often struggle with two primary issues. First, subjective label noise: unlike object detection, music choice is highly personal, and datasets often contain inconsistent pairings from amateur creators. Second, uniform importance: most models treat every frame of a video as equally relevant to the music choice, whereas in reality, a specific three-second action or "vibe" usually dictates the soundtrack. SSVMR addresses these challenges through a robust self-training framework and a focus on visual saliency.
The SSVMR Framework
SSVMR builds upon a standard retrieval backbone (CBVMR) and introduces three innovative modules designed to improve data quality and feature extraction.
1. Semi-Supervised Self-Training with R-Drop
To mitigate the impact of noisy labels, SSVMR treats the problem as a semi-supervised task. The framework uses a Gaussian Mixture Model (GMM) to analyze the loss distribution of the training data. Samples with high loss are identified as likely "noisy" labels. Instead of discarding them, the model generates "sharpened" soft labels based on its own predictions, effectively self-correcting the dataset. To maintain training stability, the authors incorporate R-Drop, a consistency regularization technique that ensures the model produces similar outputs even when different dropout neurons are activated.
2. Saliency-based Span Mixup
To overcome the problem of uniform importance, the authors propose a method to identify which parts of a video are most "salient" (important for the prediction) using gradient-based analysis. They then perform a "Span Mixup": taking the least important segment of one video and replacing it with the most important segment of another. This technique forces the model to focus on specific visual cues and learn the "locality" of the video-music relationship rather than relying on a global average of the entire clip.
3. Back Retrieval
Inspired by back-translation in Natural Language Processing, the authors implement Back Retrieval. They use a music-to-video model to find the top three most similar videos for a given song in the training set. By randomly selecting one of these videos to create a new, augmented training pair, they effectively expand the dataset with high-confidence synthetic labels.
Experimental Setup and Results
Datasets
The model was trained and evaluated using two key datasets:
- HIMV-50K: A large, heterogeneous training set of 50,000 YouTube video-music pairs, known for its high level of noise.
- MVD (Music Video Dataset): A high-quality dataset of 2,212 professional music videos used to benchmark the model's performance on clean data.
Performance
SSVMR achieved a new state-of-the-art (SOTA) on the MVD dataset, significantly outperforming previous benchmarks:
- Recall@1 (R@1): The model reached 4.45, a 34.8% relative improvement over the previous SOTA of 3.30.
- Recall@25: The model achieved 24.50, compared to the previous best of 18.20.
Ablation studies confirmed that each module contributed to this success. Removing the Back Retrieval module led to a 0.55% drop in R@1, while removing the Saliency Mixup resulted in a 1.30% decrease, proving the importance of focusing on salient visual signals.
Conclusion
SSVMR demonstrates that addressing label noise and visual saliency is key to advancing video-music retrieval. By treating noise as a distribution to be modeled rather than just a problem to be ignored, and by focusing on the most impactful moments within a video, SSVMR sets a new standard for the field. Future research may involve integrating large-scale pre-trained foundation models (like CLIP) and exploring more complex many-to-many relationships between visual content and musical genres.
Summary updated