Content-based feature exploration for transparent music recommendation using self-attentive genre classification

Seungjin Lee, Juheon Lee, Kyogu LeearXiv 2018 · OpenReview

Abstract

Interpretation of retrieved results is an important issue in music recommender systems, particularly from a user perspective. In this study, we investigate the methods for providing interpretability of content features using self-attention. We extract lyric features with the self-attentive genre classification model trained on 140,000 tracks of lyrics. Likewise, we extract acoustic features using the acoustic model with self-attention trained on 120,000 tracks of acoustic signals. The experimental results show that the proposed methods provide the characteristics that are interpretable in terms of both lyrical and musical contents. We demonstrate this by visualizing the attention weights, and by presenting the most similar songs found using lyric or audio features.

Introduction

Most music recommender systems operate as "black boxes." While they might suggest a song a user enjoys, they rarely explain why beyond high-level metadata like genre or artist similarity. This lack of transparency can hinder user trust. Existing content-based methods often provide "high-level" explanations (e.g., "this song is Jazz"). However, for a recommendation to be truly transparent, it needs "low-level" explanations. For example, a system should be able to say, "I recommended this song because it uses similar aggressive lyrical themes as your favorites" or "because the vocal timbre in the first ten seconds matches your taste." The challenge lies in extracting these interpretable features from raw audio and text data.

Methods

The proposed approach employs a Self-Attentive Genre Classification architecture. The core idea is that the same features used to classify a song's genre can serve as the basis for explaining its recommendation.

The Self-Attentive Genre Classification architecture using BiLSTM and self-attention layers for lyric and audio processing.

Feature Extraction

  • Lyrics: Songs are split into words and converted into a 500x128 matrix using pre-trained Word2Vec embeddings.
  • Audio: 30-second audio clips are converted into mel-spectrograms and processed via a pre-trained Vggish model to create a 30x128 matrix, where each row represents a one-second slice.

Model Architecture

Both modalities utilize a Bidirectional Long Short-Term Memory (BiLSTM) network to capture sequential patterns. The "secret sauce" is the Self-Attention Layer, which assigns weights to different parts of the input (specific words or specific seconds of audio). These weights indicate which parts of the song the model found most "important" for determining its genre. After training, the model produces a 2,000-dimensional content embedding used for similarity retrieval.

Datasets

The study utilized two primary datasets:

  • Lyrics Dataset: A subset of 140,000 tracks from MetroLyrics, categorized into 10 genre classes. Tracks with fewer than 70 words were excluded.
  • Audio Dataset: A subset of 120,000 tracks from the Top MAGD dataset, featuring 13 genre classes. Tracks shorter than 30 seconds were excluded.

Key Insights and Results

The model demonstrates that attention weights can serve as a powerful visualization tool for human users.

Interpretability through Attention

The model learned to "attend" to different features based on genre. In hip-hop, it focused on slang and rhymes. In metal, it focused on aggressive or religious terminology (e.g., "demon," "evil," "soul"). For audio, the model showed a strong tendency to focus on the beginning of vocal sections. For instance, in "Fix You" by Coldplay, the attention spikes exactly when the singing starts (around 13-14 seconds).

Semantic vs. Acoustic Similarity

The retrieval results highlighted the difference between lyrical and acoustic focus. Lyric-based similarity often finds "cover songs" or songs with shared niche vocabulary. Audio-based similarity, conversely, focuses on vocal timbre and instrumentation.

Comparison of most similar tracks retrieved using self-attention on lyrics versus audio features.

As shown in the retrieval results, the lyric model successfully identified songs with similar moods and key phrases, while the audio model identified tracks with similar "exciting dance" vibes and female vocal characteristics for a Rihanna query.

Conclusion

This research demonstrates that self-attention mechanisms can provide "low-level" explanations for music recommendations, moving beyond vague labels to pinpointing specific words or sounds. Future work will involve investigating performance using paired datasets where both lyrics and audio are available for the same tracks, and conducting formal user studies to confirm if these explanations increase user trust and satisfaction.

Summary updated

Abstract

Interpretation of retrieved results is an important issue in music recommender systems, particularly from a user perspective. In this study, we investigate the methods for providing interpretability of content features using self-attention. We extract lyric features with the self-attentive genre classification model trained on 140,000 tracks of lyrics. Likewise, we extract acoustic features using the acoustic model with self-attention trained on 120,000 tracks of acoustic signals. The experimental results show that the proposed methods provide the characteristics that are interpretable in terms of both lyrical and musical contents. We demonstrate this by visualizing the attention weights, and by presenting the most similar songs found using lyric or audio features.
BibTeX
@misc{lee2018content,
  author = {Seungjin Lee and Juheon Lee and Kyogu Lee},
  title = {{Content-based feature exploration for transparent music recommendation using self-attentive genre classification}},
  year = {2018},
  eprint = {1808.10600},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1808.10600},
}
Figure