Music Era Recognition Using Supervised Contrastive Learning and Artist Information

Qiqi He, Xuchen Song, Weituo Hao, Ju-Chiang Wang, Wei-Tsung Lu, Wei LiarXiv 2024 · DOI · arXiv

Abstract

Researchers from ByteDance and Fudan University have developed a novel deep learning framework to predict the release era of a song using audio mel-spectrograms and artist biographies. By combining Supervised Contrastive Learning (SCL) with a multimodal approach that fuses audio features and text, their model achieves over 90% accuracy in decade-level classification. This approach significantly outperforms standard neural networks by effectively navigating the blurry boundaries between adjacent years and mitigating the impact of data imbalance.

Introduction

Identifying the release era of a song is a complex task for machine learning models. While humans can often distinguish between the "60s sound" and the "90s sound" based on instrumentation and production styles, computers face three primary challenges:

  1. Temporal Granularity: Music evolves continuously; songs from 1989 and 1991 often share more similarities than differences, making sharp classification difficult.
  2. Metadata Scarcity: While official streaming platforms maintain release dates, user-generated content (e.g., on TikTok or YouTube) frequently lacks this information due to remixes, covers, or edits.
  3. Data Imbalance: Music datasets are heavily skewed toward the last two decades, providing far more training examples for modern music than for historical recordings.
Song distribution across eras showing significant data imbalance in the dataset.

Methodology

The researchers proposed a progression of models to address these challenges, culminating in a multimodal architecture.

Audio-Only Baselines

  • Audio-CNN: A standard Convolutional Neural Network baseline that takes mel-spectrograms as input and predicts release years using cross-entropy loss.
  • Audio-SUC (Supervised Contrastive Learning): This model replaces standard classification with a contrastive loss. It learns to pull embeddings of songs from the same era closer together in a latent space while pushing songs from different eras further apart. This spatial relationship better represents the continuous evolution of music.

Multimodal Contrastive Learning (AudioArt-MMC)

The flagship model, AudioArt-MMC, incorporates artist biographies alongside audio data. Biographies are encoded using Sentence-BERT and combined with audio features via a Transformer-based fusion module.

The proposed multimodal architecture (left) and the contrastive learning framework (right) utilizing EC and MMC losses.

To ensure the model learns a robust relationship between an artist's style and their active era, the team introduced "text-shuffle" data augmentation. By pairing audio with incorrect biographies to create negative examples, the model is forced to focus on the genuine stylistic markers of the era.

Datasets and Evaluation

The models were evaluated on three primary sources:

  • Million Song Dataset (MSD): A benchmark of ~85,000 tracks with verified release years.
  • In-House Dataset: A large-scale collection of approximately 800,000 songs.
  • AllMusic: Used to source biography text for roughly 200,000 artists.

Success was measured using $ACC_x$, a metric that counts a prediction as correct if it falls within a tolerance of $x$ years from the actual release date.

Key Results and Insights

  • The Multimodal Advantage: Adding artist information provided a 9% accuracy boost over audio-only models. Artist biographies act as a powerful proxy for time, containing cultural and technological context that anchors the audio features.
  • Decade-Level Precision: The model achieved over 90% accuracy in decade-level classification. On the more difficult 64-year classification task, the multimodal model reached 65% accuracy when allowed a 3-year tolerance.
  • Handling Imbalance: The MultiModal Contrastive (MMC) loss proved particularly effective for pre-2000 music, where data is traditionally scarce, by leveraging the rich context found in artist histories.

Limitations and Future Work

Despite its success, the model faces limitations regarding biography availability for indie or underground artists. Additionally, "cover songs" remain a challenge, as modern recordings of old compositions blend contemporary production with historical styles. Future research aims to incorporate more granular metadata, such as instrumentation tags and mood data, into a unified recognition framework.

Summary updated

Abstract

Does popular music from the 60s sound different than that of the 90s? Prior study has shown that there would exist some variations of patterns and regularities related to instrumentation changes and growing loudness across multi-decadal trends. This indicates that perceiving the era of a song from musical features such as audio and artist information is possible. Music era information can be an important feature for playlist generation and recommendation. However, the release year of a song can be inaccessible in many circumstances. This paper addresses a novel task of music era recognition. We formulate the task as a music classification problem and propose solutions based on supervised contrastive learning. An audio-based model is developed to predict the era from audio. For the case where the artist information is available, we extend the audio-based model to take multimodal inputs and develop a framework, called MultiModal Contrastive (MMC) learning, to enhance the training. Experimental result on Million Song Dataset demonstrates that the audio-based model achieves 54% in accuracy with a tolerance of 3-years range; incorporating the artist information with the MMC framework for training leads to 9% improvement further.
BibTeX
@misc{he2024music,
  author = {Qiqi He and Xuchen Song and Weituo Hao and Ju-Chiang Wang and Wei-Tsung Lu and Wei Li},
  title = {{Music Era Recognition Using Supervised Contrastive Learning and Artist Information}},
  year = {2024},
  eprint = {2407.05368},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2407.05368},
}
Figure