Clustering Music Recordings by Their Keys

Yuxiang Liu, Ye Wang, Arun Shenoy, Wei-Ho Tsai, Lianhong Cai

Introduction

Identifying the musical key of a recording is a cornerstone of Music Information Retrieval (MIR), supporting tasks from automated song recommendation to structural analysis. However, automatic key estimation remains challenging in polyphonic audio due to the presence of multiple simultaneous notes, percussion, and vocals. Traditional supervised methods often rely on extensive labeled datasets and rigid templates that struggle to generalize across diverse musical styles.

This research proposes a shift in perspective: instead of naming keys directly, can we first group recordings by their inherent tonal relationships? The authors introduce an unsupervised framework that clusters music recordings by key without prior training or labeled data, providing a robust foundation for subsequent identification.

Methodology

The proposed pipeline follows a multi-stage unsupervised approach to transform raw audio into meaningful clusters.

The multi-stage unsupervised clustering pipeline, from feature extraction to cluster estimation.
  1. Chroma-based Feature Extraction: Audio is converted into a "chromagram," a 12-bin representation where each bin corresponds to a semitone in an octave. This reduces complex polyphonic audio to its core harmonic content.
  2. Dual-Distance Metric: To quantify dissimilarity between recordings, the system combines two metrics:
  3. Symmetric Kullback-Leibler (SKL) Divergence: A statistical measure comparing pitch distributions.
  4. Center of Effect (CE): A geometric model that calculates a "tonal center" in 3D space based on the Spiral Array model.
  5. Agglomerative Clustering: Recordings are iteratively merged based on similarity.
  6. Automatic Cluster Estimation: The system determines the optimal number of clusters (up to 24) by minimizing the estimated Rand Index, allowing it to guess the number of keys present in a dataset autonomously.

Experimental Setup

The framework was evaluated using a dataset of 91 pop songs collected from CD recordings. This dataset is particularly challenging as it includes full instrumental accompaniments and vocals, unlike cleaner MIDI-based datasets. The songs represented 21 out of the 24 possible major and minor keys, with ground truth verified via commercial sheet music.

Key Results and Discussion

The system demonstrated significant stability compared to rule-based algorithms. While traditional systems often see performance degradation as dataset size increases, this unsupervised clustering approach remained consistent.

Performance metrics (Purity and Rand Index) relative to the number of clusters, highlighting the estimation of the actual number of keys.

Key findings include:

  • Cluster Purity: The system achieved an average purity of 57.3%. This increased to 70% when accounting for "Relative Key" pairs (e.g., C Major and A Minor), which share identical note sets and are inherently difficult for chroma-based algorithms to distinguish.
  • Key Estimation: The system predicted 24 clusters for a dataset containing 21 actual keys, demonstrating a high degree of accuracy in unsupervised structural estimation.
  • Geometric vs. Statistical Synergy: Combining SKL and CE allowed the system to capture both the set of notes used and the hierarchy of their importance, mitigating common errors like confusing the tonic with the dominant.

Conclusion and Future Work

The primary challenge remains the ambiguity between relative major and minor keys. However, the stability of this clustering framework suggests it is a powerful preprocessing step for large-scale music databases. Future work aims to integrate this unsupervised grouping with existing key identification algorithms, leveraging the high cross-correlation within clusters to improve the final accuracy of specific key labeling.

Summary updated

BibTeX
@inproceedings{liu2008clustering,
  author = {Yuxiang Liu and Ye Wang and Arun Shenoy and Wei-Ho Tsai and Lianhong Cai},
  title = {{Clustering Music Recordings by Their Keys}},
  booktitle = {ISMIR},
  year = {2008},
}
Figure