Cross-Modal Variational Auto-Encoder for Content-Based Micro-Video Background Music Recommendation

Jing Yi, Yaochen Zhu, Jiayi Xie, Zhenzhong Chen

TL;DR

Researchers have developed CMVAE (Cross-modal Variational Auto-encoder), a generative framework that recommends background music for micro-videos by aligning visual and textual content with audio in a shared latent space. By using "cross-generation"—forcing the model to reconstruct video features from music embeddings and vice versa—and a noise-robust fusion strategy, the system significantly outperforms existing methods on a newly released, large-scale TikTok dataset called TT-150k.

What Problem Does This Paper Address?

Micro-videos (like those on TikTok or Kwai) rely heavily on background music (BGM) to convey emotion and drive engagement. However, manually picking the right track from a library of millions is a "painstaking task."

Existing automated recommendation systems face three major hurdles:

  1. The Semantic Gap: Visual/textual data and audio data are fundamentally different. Aligning them requires more than just simple similarity matching.
  2. Data Scarcity: There are few publicly available, large-scale datasets that link micro-videos to their actual BGM.
  3. Noisy/Missing Modalities: Micro-videos often have "messy" data—shaky visuals or missing descriptions. Most models struggle when one of these inputs (like text) is absent.

Methods: The CMVAE Architecture

The authors propose CMVAE, a hierarchical Bayesian generative model. Its architecture rests on two pillars:

  • Cross-Generation: Standard models try to make video and music embeddings "close" to each other. CMVAE goes further: it constrains the latent space such that the music latent variable ($z_m$) must be able to reconstruct the video's visual and textual features, and the video latent variable ($z_v$) must reconstruct the music features. This "cross-modal imagination" forces the model to capture deep semantic correlations.
  • Product-of-Experts (PoE) Fusion: To handle the multi-modal nature of video (visual + text), the model uses the PoE principle. Instead of just concatenating features, it weights each modality based on its estimated variance (uncertainty). If a video has no description, the model automatically gives more weight to the visual "expert," making it highly robust to missing data.
The CMVAE architecture, illustrating multimodal feature extraction and the cross-modal generation process.

The model is trained using a joint objective function that combines the Evidence Lower Bound (ELBO) (for reconstruction and regularization) with a bi-directional ranking loss to ensure matched pairs are closer in the latent space than unmatched ones.

The TT-150k Dataset

The authors introduced TT-150k, a new benchmark for the field, crawled from real-world TikTok data. It includes 150,000 micro-videos associated with ~3,000 unique background music clips.

Unlike previous datasets that might have a 1:1 video-to-music ratio, TT-150k reflects real-world popularity distributions. As shown in the distribution analysis, the dataset follows a long-tail pattern where a small number of songs are extremely popular while the majority are used sparingly.

Distribution of music popularity in the TT-150k dataset, highlighting the long-tail nature of real-world BGM usage.

Key Results

The authors compared CMVAE against several baselines, including Canonical Correlation Analysis (CCA) and other VAE variants (Dual-VAE, Cross-VAE).

  • Quantitative Performance: In "Weak Generalization" tests (recommending existing music to new videos), CMVAE achieved a Recall@10 of 0.2606, nearly doubling the performance of the best non-generative deep learning baseline (CEVAR at 0.1262).
  • Strong Generalization: Even when recommending entirely new music the model had never seen during training, CMVAE maintained a Recall@10 of 0.1610, outperforming all other tested models.
  • Modality Robustness: When the textual modality was completely removed during testing, the performance drop was minimal, confirming the effectiveness of the PoE fusion strategy in handling noisy or missing data.

Limitations and Future Work

While CMVAE represents a significant step forward, the authors acknowledge that the model does not yet explicitly account for user preference bias (the "Matthew effect"). Future work may focus on temporal alignment—matching specific musical beats to video actions—and integrating personalized user history into the recommendation engine.

Summary updated

Abstract

In this paper, we propose a cross-modal variational auto-encoder (CMVAE) for content-based micro-video background music recommendation. CMVAE is a hierarchical Bayesian generative model that matches relevant background music to a micro-video by projecting these two multimodal inputs into a shared low-dimensional latent space, where the alignment of two corresponding embeddings of a matched video-music pair is achieved by cross-generation. Moreover, the multimodal information is fused by the product-of-experts (PoE) principle, where the semantic information in visual and textual modalities of the micro-video are weighted according to their variance estimations such that the modality with a lower noise level is given more weights. Therefore, the micro-video latent variables contain less irrelevant information that results in a more robust model generalization. Furthermore, we establish a large-scale content-based micro-video background music recommendation dataset, TT-150k, composed of extracted features from approximately 3,000 different background music clips associated to 150,000 micro-videos from different users. Extensive experiments on the established TT-150k dataset demonstrate the effectiveness of the proposed method. A qualitative assessment of CMVAE by visualizing some recommendation results is also included.
BibTeX
@misc{yi2021cross,
  author = {Jing Yi and Yaochen Zhu and Jiayi Xie and Zhenzhong Chen},
  title = {{Cross-Modal Variational Auto-Encoder for Content-Based Micro-Video Background Music Recommendation}},
  year = {2021},
  eprint = {2107.07268},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2107.07268},
}
Figure