On the Role of Visual Context in Enriching Music Representations

Kleanthis Avramidis, Shanti Stewart, Shrikanth Narayanan

Abstract

Human perception and experience of music is highly context-dependent. Contextual variability contributes to differences in how we interpret and interact with music, challenging the design of robust models for information retrieval. Incorporating multimodal context from diverse sources provides a promising approach toward modeling this variability. In this study, we propose VCMR — Video-Conditioned Music Representations, a contrastive learning framework that learns music representations from audio and the accompanying music videos. The contextual visual information enhances representations of music audio, as evaluated on the downstream task of music tagging. Experimental results show that the proposed framework contributes additive robustness to audio representations and indicates the extent to which musical elements are affected by visual context.

Introduction

Music Information Retrieval (MIR) systems traditionally rely almost exclusively on audio signals. However, audio is often "noisy" and subjective; identifying a specific instrument in a dense mix or discerning nuanced moods like "nostalgia" can be difficult from waveforms alone. Human perception of music is inherently multimodal, often associating sounds with visual circumstances. Music videos provide a curated source of this context, but manually labeling them at scale is impractical. This paper addresses this gap by using Self-Supervised Learning (SSL) to automatically extract relationships between audio and visual context without requiring human-annotated labels.

The VCMR Framework

The proposed Video-Conditioned Music Representations (VCMR) framework operates in three distinct stages to enrich audio embeddings with visual context.

The three stages of the VCMR framework: (1) Audio-only pre-training with distortions, (2) Multimodal contrastive pre-training aligning audio and video, and (3) Supervised fine-tuning for downstream tagging.
  1. Audio-Only Pre-training: The model learns fundamental audio features using a SimCLR-style approach, comparing distorted versions of the same audio clip (e.g., noise addition or pitch shifting) using a SampleCNN architecture.
  2. Multimodal Contrastive Pre-training: This core stage aligns audio and video segments in a shared embedding space using a contrastive loss (NT-Xent). Visual features are extracted using a pre-trained CLIP model and processed by an LSTM to capture temporal context.
  3. Supervised Fine-tuning: The pre-trained audio backbone is frozen or tuned on specific downstream tasks, such as music tagging for genre, instrument, or mood detection.

Experimental Results and Insights

The VCMR framework was evaluated on two major datasets: MagnaTagATune (MTAT) and MTG-Jamendo (MTG-M). The multimodal approach significantly outperformed audio-only baselines. On MTAT, VCMR achieved an 89.1% ROC-AUC compared to 77.4% for the audio-only model. Similarly, on MTG-M, performance rose from 57.8% to 70.7% ROC-AUC.

Data Efficiency and Robustness

A key finding of the study is the remarkable data efficiency provided by visual conditioning. VCMR models trained on only a small fraction of labeled data can match or exceed the performance of audio-only models trained on the full dataset.

ROC-AUC performance on MTAT and MTG-M datasets across varying percentages of labeled training data. VCMR (solid lines) consistently outperforms audio-only baselines (dashed lines), especially in low-data regimes.

As shown in the figure above, VCMR trained on just 10% of labeled data performs nearly as well as the audio-only model trained on 100% of the data. This suggests that visual pre-training provides a significantly stronger initialization for the audio encoder.

Visuals as a "Disentangler"

The model showed the largest performance gains in identifying instruments and vocals. The authors hypothesize that visual cues—such as seeing an instrument played on screen—help the network "disentangle" specific sounds from complex audio mixtures. Furthermore, while audio-only models performed better on very short snippets, VCMR benefited from longer input resolutions (up to 6.15 seconds), suggesting that visual context helps integrate information over longer time scales.

Conclusion

VCMR demonstrates that visual context from music videos can significantly enrich audio representations, leading to more robust and data-efficient music tagging models. While it currently trails some fully supervised state-of-the-art models, the self-supervised nature of VCMR allows it to leverage vast amounts of unlabelled video data. Future work will explore explicit feature investigation and whether object detection (e.g., identifying a guitar) can further enhance instrument labeling.

Summary updated

Abstract

Human perception and experience of music is highly context-dependent. Contextual variability contributes to differences in how we interpret and interact with music, challenging the design of robust models for information retrieval. Incorporating multimodal context from diverse sources provides a promising approach toward modeling this variability. Music presented in media such as movies and music videos provide rich multimodal context that modulates underlying human experiences. However, such context modeling is underexplored, as it requires large amounts of multimodal data along with relevant annotations. Self-supervised learning can help address these challenges by automatically extracting rich, high-level correspondences between different modalities, hence alleviating the need for fine-grained annotations at scale. In this study, we propose VCMR -- Video-Conditioned Music Representations, a contrastive learning framework that learns music representations from audio and the accompanying music videos. The contextual visual information enhances representations of music audio, as evaluated on the downstream task of music tagging. Experimental results show that the proposed framework can contribute additive robustness to audio representations and indicates to what extent musical elements are affected or determined by visual context.
BibTeX
@inproceedings{avramidis2023role,
  author = {Kleanthis Avramidis and Shanti Stewart and Shrikanth Narayanan},
  title = {{On the Role of Visual Context in Enriching Music Representations}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2023},
  url = {https://openreview.net/forum?id=r4bdQEgAVh},
}
Figure