On the Role of Visual Context in Enriching Music Representations

Kleanthis Avramidis, Shanti Stewart, Shrikanth Narayanan

This paper, presented by researchers at the University of Southern California, explores how the visual "storytelling" of music videos can be used to build better machine-learning models for understanding music. By using self-supervised learning to align audio features with their corresponding visual context, the authors demonstrate significant improvements in automated music tagging.

TL;DR

The authors introduce Video-Conditioned Music Representations (VCMR), a framework that uses contrastive self-supervised learning to enrich audio embeddings with visual context from music videos. By aligning audio with CLIP-based video features, the model achieves a double-digit percentage boost in music tagging accuracy and shows remarkable robustness when labeled training data is scarce.

What Problem Does This Paper Address?

Music Information Retrieval (MIR) systems—the tech behind how apps identify or recommend songs—traditionally rely almost exclusively on audio signals. However, audio is "noisy" and subjective. Identifying a specific instrument in a dense mix or discerning a nuanced mood (like "nostalgia" vs. "grief") is difficult from waveforms alone.

Human perception of music is inherently multimodal; we often associate sounds with the social or visual circumstances in which we hear them. Music videos provide a curated, high-quality source of this context. The challenge is that manually labeling millions of videos is impossible. This paper addresses the gap by using Self-Supervised Learning (SSL) to automatically extract these relationships without needing human-annotated labels.

Methods

The VCMR framework operates in three distinct stages:

  1. Audio-Only Pre-training: Using a SimCLR-style approach, the model learns basic audio features by comparing different "distorted" versions of the same audio clip (e.g., adding noise or shifting pitch). It uses a SampleCNN architecture to process raw waveforms.
  2. Multimodal Contrastive Pre-training: This is the core innovation. The model is presented with pairs of audio and video segments. It uses a contrastive loss (NT-Xent) to pull the embeddings of matching audio and video closer together in a shared mathematical space while pushing non-matching pairs apart.
  3. Visual Features: Instead of training a video encoder from scratch, they use CLIP (a powerful pre-trained vision-language model) to extract high-level features from video frames, which are then processed by an LSTM to capture temporal context.
  4. Supervised Fine-tuning: The pre-trained audio "backbone" is then frozen or tuned on specific downstream tasks like music tagging (genre, instrument, or mood detection).

Datasets

  • Pre-training: The authors curated a subset of the YouTube-music-video-5M dataset. They filtered it down to ~20,000 official music videos (roughly 700 hours of content), specifically removing "lyric videos" or still-image uploads to ensure rich visual context.
  • Evaluation:
    • MagnaTagATune (MTAT): 25,000 clips with 50 semantic tags (instruments, genres, moods).
  • MTG-Jamendo (MTG-M): A dataset focused on 57 "mood and theme" annotations.

Key Insights and Ideas

  • Visuals as a "Disentangler": The model showed the largest performance gains in identifying instruments and vocals. The authors hypothesize that seeing an instrument played on screen helps the network "disentangle" that specific sound from a complex audio mixture.
  • Context over Content: Unlike previous works that focus on cross-modal retrieval (finding a video for a song), VCMR uses the video to condition the audio representation itself. The goal isn't to remember the video, but to let the video "teach" the audio encoder what features are semantically important.
  • Scale and Resolution: Interestingly, the audio-only model performed better on very short snippets, while the VCMR model benefited from longer input resolutions (up to 6.15 seconds). This suggests that visual context helps the model integrate information over longer time scales.

Key Results

The multimodal approach (VCMR) significantly outperformed the audio-only baseline across the board:

  • MagnaTagATune: VCMR achieved an 89.1% ROC-AUC, a massive jump from the 77.4% achieved by the audio-only version.
  • MTG-Jamendo: Performance rose from 57.8% to 70.7% ROC-AUC.
  • Data Efficiency: One of the most striking results was that VCMR trained on only 10% of labeled data performed nearly as well as the audio-only model trained on 100% of the data. This suggests that the visual pre-training provides a much stronger "starting point" for the model.

Limitations and Future Work

  • Comparison to Supervised SOTA: While VCMR beats other self-supervised methods (like CLMR), it still slightly trails the best-performing fully supervised models (like HarmonicCNN). This is likely due to the relatively small size of the pre-training dataset (700 hours vs. the thousands of hours used in some industry models).
  • Explicit Feature Investigation: The authors note that they haven't yet isolated which specific visual elements (e.g., color, motion, or specific objects) contribute most to the performance gain.
  • Future Directions: The team plans to explore cross-modal retrieval and investigate whether explicit object detection (e.g., "there is a guitar on screen") could further boost the model's ability to label instruments.

Summary updated

Abstract

Human perception and experience of music is highly context-dependent. Contextual variability contributes to differences in how we interpret and interact with music, challenging the design of robust models for information retrieval. Incorporating multimodal context from diverse sources provides a promising approach toward modeling this variability. Music presented in media such as movies and music videos provide rich multimodal context that modulates underlying human experiences. However, such context modeling is underexplored, as it requires large amounts of multimodal data along with relevant annotations. Self-supervised learning can help address these challenges by automatically extracting rich, high-level correspondences between different modalities, hence alleviating the need for fine-grained annotations at scale. In this study, we propose VCMR – Video-Conditioned Music Representations, a contrastive learning framework that learns music representations from audio and the accompanying music videos. The contextual visual information enhances representations of music audio, as evaluated on the downstream task of music tagging. Experimental results show that the proposed framework can contribute additive robustness to audio representations and indicates to what extent musical elements are affected or determined by visual context.1
BibTeX
@inproceedings{avramidis2023role,
  author = {Kleanthis Avramidis and Shanti Stewart and Shrikanth Narayanan},
  title = {{On the Role of Visual Context in Enriching Music Representations}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2023},
  url = {https://openreview.net/forum?id=r4bdQEgAVh},
}

Loading related content...

Figure