Abstract
Researchers from ByteDance and Fudan University have developed a novel deep learning framework to predict the release era of a song using audio mel-spectrograms and artist biographies. By combining Supervised Contrastive Learning (SCL) with a multimodal approach that fuses audio features and text, their model achieves over 90% accuracy in decade-level classification. This approach significantly outperforms standard neural networks by effectively navigating the blurry boundaries between adjacent years and mitigating the impact of data imbalance.
Introduction
Identifying the release era of a song is a complex task for machine learning models. While humans can often distinguish between the "60s sound" and the "90s sound" based on instrumentation and production styles, computers face three primary challenges:
- Temporal Granularity: Music evolves continuously; songs from 1989 and 1991 often share more similarities than differences, making sharp classification difficult.
- Metadata Scarcity: While official streaming platforms maintain release dates, user-generated content (e.g., on TikTok or YouTube) frequently lacks this information due to remixes, covers, or edits.
- Data Imbalance: Music datasets are heavily skewed toward the last two decades, providing far more training examples for modern music than for historical recordings.
Methodology
The researchers proposed a progression of models to address these challenges, culminating in a multimodal architecture.
Audio-Only Baselines
- Audio-CNN: A standard Convolutional Neural Network baseline that takes mel-spectrograms as input and predicts release years using cross-entropy loss.
- Audio-SUC (Supervised Contrastive Learning): This model replaces standard classification with a contrastive loss. It learns to pull embeddings of songs from the same era closer together in a latent space while pushing songs from different eras further apart. This spatial relationship better represents the continuous evolution of music.
Multimodal Contrastive Learning (AudioArt-MMC)
The flagship model, AudioArt-MMC, incorporates artist biographies alongside audio data. Biographies are encoded using Sentence-BERT and combined with audio features via a Transformer-based fusion module.
To ensure the model learns a robust relationship between an artist's style and their active era, the team introduced "text-shuffle" data augmentation. By pairing audio with incorrect biographies to create negative examples, the model is forced to focus on the genuine stylistic markers of the era.
Datasets and Evaluation
The models were evaluated on three primary sources:
- Million Song Dataset (MSD): A benchmark of ~85,000 tracks with verified release years.
- In-House Dataset: A large-scale collection of approximately 800,000 songs.
- AllMusic: Used to source biography text for roughly 200,000 artists.
Success was measured using $ACC_x$, a metric that counts a prediction as correct if it falls within a tolerance of $x$ years from the actual release date.
Key Results and Insights
- The Multimodal Advantage: Adding artist information provided a 9% accuracy boost over audio-only models. Artist biographies act as a powerful proxy for time, containing cultural and technological context that anchors the audio features.
- Decade-Level Precision: The model achieved over 90% accuracy in decade-level classification. On the more difficult 64-year classification task, the multimodal model reached 65% accuracy when allowed a 3-year tolerance.
- Handling Imbalance: The MultiModal Contrastive (MMC) loss proved particularly effective for pre-2000 music, where data is traditionally scarce, by leveraging the rich context found in artist histories.
Limitations and Future Work
Despite its success, the model faces limitations regarding biography availability for indie or underground artists. Additionally, "cover songs" remain a challenge, as modern recordings of old compositions blend contemporary production with historical styles. Future research aims to incorporate more granular metadata, such as instrumentation tags and mood data, into a unified recognition framework.
Summary updated