Contextual Tags for music auto-tagging

Karim M. Ibrahim, Jimena Royo-Letelier, Elena V. Epure, Geoffroy Peeters, Gaël RichardSPIRE 2020 · DOI · OpenAlex

Abstract

Music recommendation systems traditionally rely on collaborative filtering or genre-based classification, which often fail to account for the situational context of the listener. This paper introduces a new dataset of 50,000 tracks labeled with 15 contextual tags (e.g., gym, car, sleep) derived from user-created playlists. We propose an audio-based auto-tagging model using a Convolutional Neural Network (CNN) and a novel weighted loss function designed to address the "missing negative labels" problem inherent in user-generated data. Our results demonstrate that while certain moods like sad or club are highly predictable from audio signals, functional contexts like work or car remain challenging due to their subjective nature.

Introduction

Traditional music recommendation assumes that user taste is static. However, musical needs fluctuate significantly based on activity and environment. To build context-aware systems without invasive sensor tracking (like GPS or accelerometers), researchers turn to auto-tagging: training models to identify the "vibe" of a song directly from its audio signal.

A major hurdle in this field is the lack of high-quality, large-scale datasets. Most existing data is "noisy"; for instance, if a song isn't labeled "Gym," it doesn't necessarily mean it is unsuitable for a workout—it might simply lack that specific tag in the database. This paper addresses this "Missing Negative Labels" problem through a new dataset and a specialized training objective.

Methodology

The research utilizes a three-pronged approach to bridge the gap between raw audio and human context:

  1. Proxy Labeling: To scale the dataset, the authors used playlist titles as proxies. A track is associated with a tag (e.g., workout) if it appears frequently in playlists with that specific title in the Deezer catalog.
  2. CNN Architecture: The model employs a 4-layer Convolutional Neural Network trained on melspectrograms. This baseline architecture allows the study to focus on the impact of data labeling and loss function innovations.
  3. Sample-Level Weighted Cross Entropy: To handle missing labels, the researchers modified the loss function. They assigned "confidence weights" to labels: positive weights were determined via TF-IDF to prioritize strong associations, while negative weights were estimated using label co-occurrence (e.g., a song tagged as "Dance" is statistically unlikely to be a "Sleep" song).

The Deezer Contextual Dataset

The study introduces the Deezer Contextual Dataset, featuring approximately 50,000 unique tracks. Each track is labeled with 15 tags: car, chill, club, dance, gym, happy, night, party, relax, running, sad, sleep, summer, work, workout. To prevent the model from over-learning popular tags, classes were capped at 20,000 tracks each.

Key Results and Analysis

The model achieved a macro-average AUC of 0.73, but performance varied significantly across different tags.

Performance metrics across the 15 contextual tags. Note the high AUC for 'Sad' and 'Club' versus the lower performance for 'Night' and 'Work'.

As shown in the table above, contexts with distinct acoustic profiles—such as Sad (0.85 AUC), Club (0.84), and Sleep (0.84)—were the easiest to predict. Conversely, Night (0.58 AUC) and Work (0.62) performed poorly, suggesting these contexts are defined more by personal preference than by universal audio characteristics.

Impact of Weighted Loss

A critical finding was the effectiveness of the proposed weighting scheme. By treating "0" labels as "unknown" rather than definitive negatives (the "ignore zeros" approach), the model's ability to find relevant songs improved dramatically.

Comparison of different weighting schemes. The 'ignore zeros' approach significantly boosts Recall.

As illustrated in the comparison table, using negative weights while ignoring zeros increased Recall from 0.69 to 0.94. This proves that accounting for the incompleteness of user-generated labels is essential for training robust auto-tagging models.

Conclusion

Audio-based auto-tagging is a viable path for context-aware music recommendation, particularly for moods with strong acoustic signatures. However, the poor performance of functional tags like Work suggests that audio alone may not be sufficient for all contexts. Future work should explore combining audio analysis with temporal data (time of day) or user metadata to better capture the subjective nature of music usage.

Summary updated

Abstract

The dataset is composed of 15 contextual tags extracted based on user's usage through created playlists in the Deezer catalog. The tags are: " car, chill, club, dance, gym, happy, night, party, relax, running, sad, sleep, summer, work, workout". For each track one or multiple tags are associated with it indicating that users listen to the track in the associated context. The creation of the dataset and the initial baseline of an auto-tagging model is described in the paper: Ibrahim, Karim M., Jimena Royo-Letelier, Elena V. Epure, Geoffroy Peeters, and Gaël Richard. "AUDIO-BASED AUTO-TAGGING WITH CONTEXTUAL TAGS FOR MUSIC." <em>2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)</em>. IEEE, 2020. The dataset is composed of the SONG_ID which is the ID of the track in the Deezer catalog. Each track is labeled with each tag as either 1 (indicating a track's presence in the context) or 0 (indicating a track's absence). The 30 seconds track previews used to train the model in the paper can be accessed through the Deezer API: https://developers.deezer.com/api .
BibTeX
@inproceedings{ibrahim2020contextual,
  author = {Karim M. Ibrahim and Jimena Royo-Letelier and Elena V. Epure and Geoffroy Peeters and Gaël Richard},
  title = {{Contextual Tags for music auto-tagging}},
  booktitle = {SPIRE},
  year = {2020},
  doi = {10.5281/zenodo.3648287},
}

Loading related content...

Figure