Addressing the confounds of accompaniments in singer identification

Tsung-Han Hsieh, Kai-Hsiang Cheng, Zhe-Cheng Fan, Yu-Ching Yang, Yi-Hsuan Yang

Abstract

Identifying singers is an important task with many applications. However, the task remains challenging due to many issues. One major issue is related to the confounding factors from the background instrumental music that is mixed with the vocals in music production. A singer identification model may learn to extract non-vocal related features from the instrumental part of the songs, if a singer only sings in certain musical contexts (e.g., genres). The model cannot therefore generalize well when the singer sings in unseen contexts. In this paper, we attempt to address this issue. Specifically, we employ open-unmix, an open source tool with state-of-the-art performance in source separation, to separate the vocal and instrumental tracks of music. We then investigate two means to train a singer identification model: by learning from the separated vocal only, or from an augmented set of data where we "shuffle-and-remix" the separated vocal tracks and instrumental tracks of different songs to artificially make the singers sing in different contexts. We also incorporate melodic features learned from the vocal melody contour for better performance. Evaluation results on a benchmark dataset called the artist20 shows that this data augmentation method greatly improves the accuracy of singer identification.

Introduction

Singer Identification (SID) is the task of automatically identifying the artist performing a vocal track. While seemingly straightforward, SID is often hindered by the presence of background instrumental music. Most artists are associated with specific genres or production styles, leading models to inadvertently learn to identify the instrumental accompaniment or genre-specific arrangements rather than the unique vocal characteristics of the singer. This "confounding" effect results in poor generalization; a model trained on studio mixes may fail to recognize a singer in an acoustic setting or a different musical genre. This research aims to decouple vocal features from their musical context to create more robust SID systems.

Methods

The proposed solution involves a novel architecture and a specialized data augmentation strategy designed to force the model to focus on vocal-invariant features.

CRNNM Architecture

The researchers developed the CRNNM architecture, an extension of the standard Convolutional Recurrent Neural Network (CRNN). As shown in Figure 1, the model features two parallel branches. The first branch processes the standard mel-spectrogram of the audio. The second branch processes melodic features extracted via the CREPE pitch tracker. This dual-input approach allows the model to capture both raw spectral data and the stylistic nuances of the singer's melodic contour.

The CRNNM architecture incorporates both mel-spectrogram and melodic contour features through parallel convolutional blocks and GRU layers.

Shuffle-and-Remix Augmentation

To break the statistical bond between a singer and their typical accompaniment, the authors utilized Open-Unmix to separate songs into vocal and instrumental tracks. They then implemented a "shuffle-and-remix" strategy. By pairing the vocal track of one artist with the instrumental track of a different artist's song, they created a synthetic dataset where singers appear in diverse, often mismatched, musical contexts. This forces the model to identify the singer regardless of the background music.

The 'Shuffle-and-Remix' data augmentation process uses source separation to decouple vocals from their original accompaniments.

Experimental Setup

The model was evaluated using the artist20 dataset (20 artists, 1,413 tracks). To ensure rigorous testing, an album-split was used, where the model is trained on specific albums and tested on entirely different ones. This prevents the model from relying on the specific production "vibe" or recording quality of a single session.

Results and Discussion

The combination of the CRNNM architecture and the shuffle-and-remix augmentation significantly outperformed the baseline. The baseline CRNN model trained on original data achieved a song-level F1 score of 0.67, while the proposed CRNNM with data augmentation reached 0.75.

Key insights from the results include:

  • Context Invariance: Training on vocals alone was found to be less effective than the shuffle-and-remix method. The latter teaches the model what remains constant about a voice across varying backgrounds.
  • Melody as Identity: The inclusion of melodic contours proved beneficial, suggesting that the way a singer transitions between pitches is a discriminative "timbre" feature.
  • Vocalness Sensitivity: Analysis revealed a correlation (0.39) between vocal volume and model confidence. Performance drops significantly during instrumental breaks or quiet vocal passages, highlighting the model's sensitivity to the signal-to-noise ratio of the voice.

Conclusion

This work demonstrates that decoupling vocals from their musical context through source separation and remixing is a powerful technique for improving singer identification. By incorporating melodic features, the CRNNM model achieves state-of-the-art results on the artist20 benchmark. Future work will focus on refining the augmentation process to be "key-aware" and "tempo-aware" to create more musically coherent synthetic data, as well as implementing vocal activity detection to filter out non-vocal segments during inference.

Summary updated

Abstract

Identifying singers is an important task with many applications. However, the task remains challenging due to many issues. One major issue is related to the confounding factors from the background instrumental music that is mixed with the vocals in music production. A singer identification model may learn to extract non-vocal related features from the instrumental part of the songs, if a singer only sings in certain musical contexts (e.g., genres). The model cannot therefore generalize well when the singer sings in unseen contexts. In this paper, we attempt to address this issue. Specifically, we employ open-unmix, an open source tool with state-of-the-art performance in source separation, to separate the vocal and instrumental tracks of music. We then investigate two means to train a singer identification model: by learning from the separated vocal only, or from an augmented set of data where we "shuffle-and-remix" the separated vocal tracks and instrumental tracks of different songs to artificially make the singers sing in different contexts. We also incorporate melodic features learned from the vocal melody contour for better performance. Evaluation results on a benchmark dataset called the artist20 shows that this data augmentation method greatly improves the accuracy of singer identification.
BibTeX
@inproceedings{hsieh2020addressing,
  author = {Tsung-Han Hsieh and Kai-Hsiang Cheng and Zhe-Cheng Fan and Yu-Ching Yang and Yi-Hsuan Yang},
  title = {{Addressing the confounds of accompaniments in singer identification}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2020},
  url = {https://openreview.net/forum?id=gCGjkNZKKc},
}
Figure