Music Artist Classification with Convolutional Recurrent Neural Networks

Zain Nasrullah, Yue Zhao

Abstract

Previous attempts at music artist classification use frame level audio features which summarize frequency content within short intervals of time. Comparatively, more recent music information retrieval tasks take advantage of temporal structure in audio spectrograms using deep convolutional and recurrent models. This paper revisits artist classification with this new framework and empirically explores the impacts of incorporating temporal structure in the feature representation. To this end, an established classification architecture, a Convolutional Recurrent Neural Network (CRNN), is applied to the artist20 music artist identification dataset under a comprehensive set of conditions. These include audio clip length, which is a novel contribution in this work, and previously identified considerations such as dataset split and feature level. Our results improve upon baseline works, verify the influence of the producer effect on classification performance and demonstrate the trade-offs between audio length and training set size. The best performing model achieves an average F1 score of 0.937 across three independent trials which is a substantial improvement over the corresponding baseline under similar conditions. Additionally, to showcase the effectiveness of the CRNN's feature extraction capabilities, we visualize audio samples at the model's bottleneck layer demonstrating that learned representations segment into clusters belonging to their respective artists.

Introduction

Identifying the artist behind a piece of music is a core task in Music Information Retrieval (MIR). While humans can often identify a favorite artist even when hearing a new song, algorithmic approaches have historically struggled due to two main challenges: feature limitations and the "producer effect."

Traditional methods rely on Mel-frequency cepstral coefficients (MFCCs), which summarize frequency content but often lose the "temporal flow" or rhythm that defines an artist's style. Furthermore, models often "cheat" by learning the specific production qualities of an album (the producer effect) rather than the artist's actual musical style. If a model is tested on songs from the same album it was trained on, its performance is artificially inflated. This paper explores a deep learning framework designed to overcome these hurdles.

Methodology: The CRNN Architecture

The researchers adapted a Convolutional Recurrent Neural Network (CRNN) architecture, which combines the strengths of two different neural architectures to capture both timbral patterns and temporal structure.

The CRNN architecture used for artist classification, featuring 2D convolutional layers followed by Gated Recurrent Units (GRUs).
  • Convolutional Layers (CNN): Four 2D convolutional layers act as feature extractors, identifying patterns in frequency and local structure within log-mel spectrograms.
  • Recurrent Layers (RNN): Two Gated Recurrent Units (GRUs) follow the CNN to summarize how those patterns evolve over time.
  • Optimization: The model uses the Exponential Linear Unit (ELU) activation function for better generalization and Batch Normalization to stabilize training.

Experimental Setup

The study primarily uses the artist20 dataset, which consists of 1,413 tracks from 20 artists, with 6 albums per artist. This structure allows for a strict "album-split" to test for the producer effect.

A novel contribution of this work is the exploration of the Clip Length Trade-off. The authors experimented with audio clips ranging from 1 to 30 seconds, finding a "sweet spot" at around 3 seconds. While longer clips provide more temporal context, they result in fewer total training samples, leading to a point of diminishing returns and potential overfitting.

Key Results and Insights

The CRNN model demonstrated a significant leap over traditional machine learning baselines:

  • Top Performance: The best CRNN model achieved an average F1 score of 0.937 using 3-second clips and song-level voting, compared to the previous SVM-based baseline of 0.839.
  • Album-Split Resilience: Even in the harder album-split scenario, the CRNN achieved an F1 score of 0.674, outperforming traditional GMM and SVM methods.

Latent Space Clustering

To showcase the effectiveness of the CRNN's feature extraction, the authors visualized audio samples at the model's bottleneck layer using t-SNE.

t-SNE visualization of the model's bottleneck layer, showing distinct clusters for different artists and stylistic groupings.

The visualization demonstrates that the model naturally groups artists by style. For example, rock bands like The Beatles and Led Zeppelin clustered near each other, while pop artists like Madonna and Tori Amos formed separate neighborhoods.

Conclusion and Future Work

This research confirms that incorporating temporal structure via CRNNs significantly improves artist classification. However, limitations such as data scarcity remain; deep learning models thrive on massive data, but most artists have limited discographies. Future work may involve audio augmentation (pitch shifting, time stretching) or pre-training the model on larger tasks like genre classification before fine-tuning for specific artist identification.

Summary updated

Abstract

Previous attempts at music artist classification use frame level audio features which summarize frequency content within short intervals of time. Comparatively, more recent music information retrieval tasks take advantage of temporal structure in audio spectrograms using deep convolutional and recurrent models. This paper revisits artist classification with this new framework and empirically explores the impacts of incorporating temporal structure in the feature representation. To this end, an established classification architecture, a Convolutional Recurrent Neural Network (CRNN), is applied to the artist20 music artist identification dataset under a comprehensive set of conditions. These include audio clip length, which is a novel contribution in this work, and previously identified considerations such as dataset split and feature level. Our results improve upon baseline works, verify the influence of the producer effect on classification performance and demonstrate the trade-offs between audio length and training set size. The best performing model achieves an average F1 score of 0.937 across three independent trials which is a substantial improvement over the corresponding baseline under similar conditions. Additionally, to showcase the effectiveness of the CRNN's feature extraction capabilities, we visualize audio samples at the model's bottleneck layer demonstrating that learned representations segment into clusters belonging to their respective artists.
BibTeX
@inproceedings{nasrullah2019music,
  author = {Zain Nasrullah and Yue Zhao},
  title = {{Music Artist Classification with Convolutional Recurrent Neural Networks}},
  booktitle = {IJCNN},
  year = {2019},
  doi = {10.1109/ijcnn.2019.8851988},
}
Figure