SSVMR: Saliency-Based Self-Training for Video-Music Retrieval

Xuxin Cheng, Zhihong Zhu, Hongxiang Li, Yaowei Li, Yuexian Zou

TL;DR

SSVMR (Saliency-based Self-training for Video-Music Retrieval) is a new framework designed to match the perfect background music to a video clip. By intelligently filtering out "noisy" labels using self-training and focusing on the most important parts of a video through saliency-based mixing, it achieves a massive 34.8% relative improvement over previous state-of-the-art models.

What Problem Does This Paper Address?

With the explosion of short-form video content (TikTok, Reels, etc.), finding the right background music (BGM) has become a high-demand task. However, automating Video-Music Retrieval (VMR) is notoriously difficult for two main reasons:

  1. Subjective Label Noise: Unlike identifying a "cat" in a photo, choosing music is subjective. Different editors might pick different songs for the same clip. Furthermore, datasets often mix professional music videos with amateur user-generated content, leading to inconsistent relationships between the video and the audio.
  2. Uniform Importance: Most existing models treat every second of a video as equally important. In reality, a specific three-second "vibe" or action sequence usually dictates the music choice, while the rest of the video might be less relevant.

Methods

The authors propose SSVMR, which introduces three clever modules to the standard retrieval backbone (CBVMR):

  • Semi-Supervised Self-Training with R-Drop: To handle "noisy" labels, the model uses a Gaussian Mixture Model (GMM) to analyze loss distributions. Samples with high loss are flagged as likely "noisy." The model then replaces their original labels with "sharpened" soft labels generated by the model itself. To keep the model stable, they use R-Drop, which forces the model to produce consistent outputs even when different "dropout" neurons are active.
  • Saliency-based Span Mixup: Instead of just looking at whole videos, the authors calculate which parts of a video are most "salient" (important for the prediction) using gradients. They then create new training samples by taking the least important part of one video and replacing it with the most important part of another. This forces the model to learn exactly which visual cues trigger a specific musical match.
  • Back Retrieval: Borrowing a trick from Natural Language Processing (back-translation), the authors use a music-to-video model to find the top three most similar videos for a given song. They randomly pick one to create a new, augmented training pair, effectively expanding their dataset.

Datasets

The researchers utilized two primary datasets:

  • HIMV-50K: A large-scale training set containing 50,000 video-music pairs from YouTube. It is known for being "noisy" and heterogeneous (varying quality).
  • MVD (Music Video Dataset): A collection of 2,212 professional-quality music videos. The authors used 2,000 of these clips to evaluate how well the model performs on high-quality data.

Key Insights and Ideas

  • Noise as a Distribution: Rather than trying to manually clean a massive dataset, the authors realized that "clean" and "noisy" samples follow different loss patterns. Modeling this with a GMM allows the AI to "self-correct" its training data.
  • Saliency is the Signal: By swapping video segments based on saliency, the model learns "locality." It realizes that the music isn't just a response to the average of the video, but to specific, high-impact moments.
  • The "Back-Translation" of Video: The success of Back Retrieval suggests that in cross-modal tasks where labels are fuzzy, synthetic pairs generated by the model itself can be more useful for training than the original, potentially mismatched labels.

Key Results

The SSVMR framework outperformed all previous benchmarks on the MVD dataset:

  • Recall@1 (R@1): Achieved 4.45, compared to the previous best of 3.30 (a 34.8% relative improvement).
  • Recall@25: Reached 24.50, significantly higher than the previous state-of-the-art (18.20).
  • Ablation Studies: The researchers proved that every component mattered. Removing the "Back Retrieval" module caused a 0.55% drop in R@1, while removing the "Saliency Mixup" caused a 1.30% drop.

Limitations and Future Work

While the relative improvement is impressive, the absolute R@1 score (4.45%) highlights just how difficult video-music retrieval remains. The authors acknowledge that the matching criteria are still somewhat ambiguous.

For future work, they suggest:

  • Integrating pre-training techniques (like those used in CLIP or other large-scale foundation models) to give the model a better baseline understanding of visual and auditory concepts before it even starts the VMR task.
  • Exploring more complex ways to handle the "many-to-many" relationship between videos and music, as one video could realistically be paired with many different genres of music.

Summary updated

Abstract

With the rise of short videos, the demand for selecting appropriate background music (BGM) for a video has increased significantly, video-music retrieval (VMR) task gradually draws much attention by research community. As other cross-modal learning tasks, existing VMR approaches usually attempt to measure the similarity between the video and music in the feature space. However, they (1) neglect the inevitable label noise; (2) neglect to enhance the ability to capture critical video clips. In this paper, we propose a novel saliency-based self-training framework, which is termed SSVMR. Specifically, we first explore to fully make use of the information containing in the training dataset by applying a semi-supervised method to suppress the adverse impact of label noise problem, where a self-training approach is adopted. In addition, we propose to capture the saliency of the video by mixing two videos at span level and preserving the locality of the two original videos. Inspired by back translation in NLP, we also conduct back retrieval to obtain more training data. Experimental results on MVD dataset show that our SSVMR achieves the state-of-the-art performance by a large margin, obtaining a relative improvement of 34.8% over the previous best model in terms of R@1.
BibTeX
@inproceedings{cheng2023ssvmr,
  author = {Xuxin Cheng and Zhihong Zhu and Hongxiang Li and Yaowei Li and Yuexian Zou},
  title = {{SSVMR: Saliency-Based Self-Training for Video-Music Retrieval}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2023},
  url = {https://openreview.net/forum?id=0FTG8ltPnAw},
}

Loading related content...

Figure