Using Shared Vector Representations of Words and Chords in Music for Genre Classification

Timothy Greer, Shrikanth NarayananSMM 2019 · OpenReview

TL;DR

This paper introduces a novel method for music genre classification by learning shared vector representations for song lyrics and chord progressions. By treating lyrics and chords as two "languages" and embedding them in a common vector space, the authors demonstrate improved genre prediction, suggesting a deeper understanding of how these multimodal elements interact to shape our perception of music.

What Problem Does This Paper Address?

The paper tackles the challenge of understanding and classifying music, a complex, multimodal perceptual experience. While music genre is a key aspect of how we categorize and consume music, it's a subjective construct influenced by various elements. Existing approaches often focus on either lyrical content or musical structure (like chords) in isolation. The authors aim to bridge this gap by exploring how lyrics and chords interact and how this interaction can be leveraged for more accurate music genre classification. They pose questions like: how does musical structure affect our perception of lyrics, and vice versa? Can we better categorize a song with pop lyrics but R&B chords using a combined approach?

Methods

The core of the paper's methodology lies in adapting the "skip-gram" neural network architecture, commonly used in Natural Language Processing (NLP) for learning word embeddings, to a "bilingual" music context. Instead of just words, they learn embeddings for both lyrics and chord progressions simultaneously.

Here's a breakdown:

  • Shared Vector Space: They create a joint vocabulary of words and chords and train a model to predict the context (surrounding words/chords) for a given word or chord. This is achieved by modifying the standard skip-gram objective function to include "cross-lingual" prediction tasks: predicting chords based on lyrics and vice versa.
  • Bilingual Skip-Gram: This adaptation allows them to learn embeddings where lyrics and chords are represented in the same vector space, capturing their co-occurrence and interdependencies.
  • Genre Classification: Once these shared embeddings are learned, they are used as features for a genre classification task. For each song, they average the embeddings of all its lyrical and chordal tokens to create a song-level feature vector. This vector is then fed into a k-Nearest Neighbors (k-NN) classifier.

Datasets

  • Ukutabs: The primary dataset used for training the embeddings and for the genre classification task. This website provides a large archive of popular songs with aligned chordal and lyrical passages. The authors curated a dataset of 5,304 songs from Ukutabs, resulting in 190,165 musical segments.
  • Billboard Charts: Used as the ground truth for genre labels. The authors collected song titles from Latin, Rock, R&B/Hip-Hop, and Pop Billboard charts over the last 20 years and matched them to their Ukutabs dataset, resulting in 850 unique songs for evaluation.

Key Insights and Ideas

  • Lyrics and Chords are Intertwined: The most significant insight is that learning shared representations for lyrics and chords demonstrably improves music genre classification. This suggests that the interplay between lyrical content and harmonic structure is a crucial factor in how we perceive and categorize music.
  • Multimodal Approach is Key: The "Chords & Lyrics" model, which leverages joint embeddings, significantly outperforms models that use only chord embeddings or only lyric embeddings. This strongly supports the idea that a multimodal approach is more effective for understanding music perception.
  • NLP Techniques for Music: The paper highlights the power of adapting successful NLP techniques (like word embeddings) to the domain of music, opening up new avenues for research in music information retrieval and computational musicology.
  • Genre as a Perceptual Construct: By using genre classification as a proxy for music perception, the study underscores the subjective yet classifiable nature of musical experiences.

Key Results

The "Chords & Lyrics" model, using joint embeddings, achieved the best performance across several metrics for multi-label genre classification:

  • Exact Match Ratio (EMR): 20.4%
  • Label Accuracy: 37.9%
  • Label-based, Micro-averaged F1-score: 0.428

These results significantly outperformed baseline models (like predicting the most common genre) and monolingual embedding models (Chords Only, Lyrics Only). For instance, the "Chords & Lyrics" model achieved a label accuracy of 37.9%, compared to 18.5% for "Lyrics Only" and 16.5% for "Chords Only."

Limitations and Future Work

  • Dataset Bias: The Ukutabs dataset has a bias towards music playable by ukulele musicians, which might limit the generalizability of the findings.
  • Chord Caster Accuracy: The automated chord caster used to standardize chord types was untested and might have introduced inaccuracies in up to 4.1% of chords.
  • Audio Features: The current study relies solely on symbolic representations (lyrics and chords). The authors suggest future work should compare their symbolic approach with models that incorporate auditory features (like rhythm and harmony) to gain a deeper understanding of music perception.
  • Weighting Objective Function: The paper mentions that different weight coefficients in the objective function could be explored to emphasize certain aspects (mono-lingual vs. cross-lingual) of the embeddings.
  • Advanced Classifiers: While k-NN was used for its simplicity and speed, exploring more advanced multi-label classification models (e.g., RAKEL) could potentially yield even better results.
  • Key Estimation: The simple key estimation method used was effective but requires further investigation for broader generalization.

Summary updated

Abstract

With so much music readily available for consumption today, it has never been more important to study music perception. In this paper, we represent lyrics and chords in a shared vector space using a phrase-aligned lyrics-and-chords corpus and show that models that use these shared representations can predict musical genre of songs—a perceptual construct of music listening—better than models that do not use these representations. This work adds to our understanding of how lyrics and chords interact with one another in music and has applications in multimodal perception and music information retrieval.
BibTeX
@inproceedings{greer2019using,
  author = {Timothy Greer and Shrikanth Narayanan},
  title = {{Using Shared Vector Representations of Words and Chords in Music for Genre Classification}},
  booktitle = {SMM},
  year = {2019},
  url = {https://openreview.net/forum?id=4UGxJ17WTU},
}
Figure