Abstract
Human perception and experience of music is highly context-dependent. Contextual variability contributes to differences in how we interpret and interact with music, challenging the design of robust models for information retrieval. Incorporating multimodal context from diverse sources provides a promising approach toward modeling this variability. In this study, we propose VCMR — Video-Conditioned Music Representations, a contrastive learning framework that learns music representations from audio and the accompanying music videos. The contextual visual information enhances representations of music audio, as evaluated on the downstream task of music tagging. Experimental results show that the proposed framework contributes additive robustness to audio representations and indicates the extent to which musical elements are affected by visual context.
Introduction
Music Information Retrieval (MIR) systems traditionally rely almost exclusively on audio signals. However, audio is often "noisy" and subjective; identifying a specific instrument in a dense mix or discerning nuanced moods like "nostalgia" can be difficult from waveforms alone. Human perception of music is inherently multimodal, often associating sounds with visual circumstances. Music videos provide a curated source of this context, but manually labeling them at scale is impractical. This paper addresses this gap by using Self-Supervised Learning (SSL) to automatically extract relationships between audio and visual context without requiring human-annotated labels.
The VCMR Framework
The proposed Video-Conditioned Music Representations (VCMR) framework operates in three distinct stages to enrich audio embeddings with visual context.
- Audio-Only Pre-training: The model learns fundamental audio features using a SimCLR-style approach, comparing distorted versions of the same audio clip (e.g., noise addition or pitch shifting) using a SampleCNN architecture.
- Multimodal Contrastive Pre-training: This core stage aligns audio and video segments in a shared embedding space using a contrastive loss (NT-Xent). Visual features are extracted using a pre-trained CLIP model and processed by an LSTM to capture temporal context.
- Supervised Fine-tuning: The pre-trained audio backbone is frozen or tuned on specific downstream tasks, such as music tagging for genre, instrument, or mood detection.
Experimental Results and Insights
The VCMR framework was evaluated on two major datasets: MagnaTagATune (MTAT) and MTG-Jamendo (MTG-M). The multimodal approach significantly outperformed audio-only baselines. On MTAT, VCMR achieved an 89.1% ROC-AUC compared to 77.4% for the audio-only model. Similarly, on MTG-M, performance rose from 57.8% to 70.7% ROC-AUC.
Data Efficiency and Robustness
A key finding of the study is the remarkable data efficiency provided by visual conditioning. VCMR models trained on only a small fraction of labeled data can match or exceed the performance of audio-only models trained on the full dataset.
As shown in the figure above, VCMR trained on just 10% of labeled data performs nearly as well as the audio-only model trained on 100% of the data. This suggests that visual pre-training provides a significantly stronger initialization for the audio encoder.
Visuals as a "Disentangler"
The model showed the largest performance gains in identifying instruments and vocals. The authors hypothesize that visual cues—such as seeing an instrument played on screen—help the network "disentangle" specific sounds from complex audio mixtures. Furthermore, while audio-only models performed better on very short snippets, VCMR benefited from longer input resolutions (up to 6.15 seconds), suggesting that visual context helps integrate information over longer time scales.
Conclusion
VCMR demonstrates that visual context from music videos can significantly enrich audio representations, leading to more robust and data-efficient music tagging models. While it currently trails some fully supervised state-of-the-art models, the self-supervised nature of VCMR allows it to leverage vast amounts of unlabelled video data. Future work will explore explicit feature investigation and whether object detection (e.g., identifying a guitar) can further enhance instrument labeling.
Summary updated