Addressing The Confounds Of Accompaniments In Singer Identification

Tsung-Han Hsieh, Kai-Hsiang Cheng, Zhe-Cheng Fan, Yu-Ching Yang, Yi-Hsuan Yang

TL;DR

Researchers have developed a way to make singer identification (SID) models more robust by breaking the "bond" between a singer's voice and their typical musical style. By using source separation to "shuffle-and-remix" vocals with different instrumental tracks and incorporating melodic contours, they achieved a significant jump in identification accuracy on the benchmark artist20 dataset.

What Problem Does This Paper Address?

Identifying a singer in a recording (Singer Identification, or SID) seems straightforward, but background music creates a massive "confounding" problem. Most singers have a preferred genre or production style. If a model is trained on standard studio mixes, it might accidentally learn to identify the instruments or the genre (e.g., "this sounds like a jazz arrangement") rather than the actual vocal characteristics of the artist.

This leads to poor generalization: a model trained this way might fail to recognize a singer if they perform an acoustic version of a song or switch genres. The authors aim to force the model to focus strictly on the voice by decoupling it from the accompaniment.

Methods

The authors propose a two-pronged approach involving a novel model architecture and a clever data augmentation strategy:

  1. Source Separation: They use Open-Unmix, a state-of-the-art tool, to split songs into separate vocal and instrumental tracks.
  2. Shuffle-and-Remix Augmentation: Instead of just training on the original songs, they create a synthetic dataset. They take the vocal track from Singer A and mix it with the instrumental track from a completely different song by Singer B. This forces the model to recognize Singer A regardless of the musical context.
  3. CRNNM Architecture: They build upon a standard Convolutional Recurrent Neural Network (CRNN). Their version, called CRNNM, adds a secondary branch that processes melodic features (extracted using the CREPE pitch tracker). This branch helps the model capture the "timbre" and "style" of the melody contour alongside the raw spectral data from the mel-spectrogram.
  4. Training Logic: The model processes 5-second audio segments. For the final "per-song" prediction, it uses a majority vote across all segments in that song.

Datasets

The study uses the artist20 dataset, a standard benchmark in the field.

  • Composition: 20 artists, each with 6 albums, totaling 1,413 tracks.
  • Evaluation Split: The authors use an album-split (training on some albums and testing on others). This is much harder than a "song-split" because it prevents the model from "cheating" by memorizing the specific production quality or "vibe" of a particular recording session.

Key Insights and Ideas

  • Breaking the "Bonds": The most important insight is that simply removing the accompaniment (Vocal-only training) isn't the best solution. In fact, training on vocals alone performed worse than the original mixes in some tests. The "shuffle-and-remix" method is superior because it teaches the model what is invariant about a voice across many different backgrounds.
  • Melody as Timbre: While pitch is usually seen as "what" is being sung, the way a singer moves between pitches (the contour) carries information about their vocal technique and identity.
  • The "Vocalness" Correlation: The researchers found a correlation (0.39) between the volume of the vocal track and the model's confidence. If a 5-second segment has very quiet vocals or just an instrumental break, the model's accuracy plummets. This highlights that SID models are highly sensitive to the signal-to-noise ratio of the voice.

Key Results

The combination of the new architecture and data augmentation yielded impressive gains:

  • Baseline (CRNN on original data): Achieved a song-level F1 score of 0.67.
  • Proposed (CRNNM + Data Augmentation): Achieved a song-level F1 score of 0.75.
  • Visualization: t-SNE plots (which project complex data into 2D space) showed that the embeddings from the CRNNM model were much more clearly clustered by artist than the baseline model, proving the features learned were more discriminative.

Limitations and Future Work

  • Silence and Instrumental Gaps: The model currently struggles with segments where the singer isn't active. The authors suggest adding a "vocal detector" as a pre-filter to skip non-vocal segments during inference.
  • Musical Coherence: The current "shuffle-and-remix" method is musically "blind"—it might mix a vocal in C-major with an instrumental in F-minor at a different tempo. Future work could involve "key-aware" and "tempo-aware" remixing to create more natural-sounding training data.
  • Architecture Refinement: The authors plan to experiment with replacing the convolutional layers in the melody branch with more GRUs, as melodic contours are essentially time-series data.

Summary updated

Abstract

Identifying singers is an important task with many applications. However, the task remains challenging due to many issues. One major issue is related to the confounding factors from the background instrumental music that is mixed with the vocals in music production. A singer identification model may learn to extract non-vocal related features from the instrumental part of the songs, if a singer only sings in certain musical contexts (e.g., genres). The model cannot therefore generalize well when the singer sings in unseen contexts. In this paper, we attempt to address this issue. Specifically, we employ open-unmix, an open source tool with state-of-the-art performance in source separation, to separate the vocal and instrumental tracks of music. We then investigate two means to train a singer identification model: by learning from the separated vocal only, or from an augmented set of data where we "shuffle-and-remix" the separated vocal tracks and instrumental tracks of different songs to artificially make the singers sing in different contexts. We also incorporate melodic features learned from the vocal melody contour for better performance. Evaluation results on a benchmark dataset called the artist20 shows that this data augmentation method greatly improves the accuracy of singer identification.
BibTeX
@inproceedings{hsieh2020addressing,
  author = {Tsung-Han Hsieh and Kai-Hsiang Cheng and Zhe-Cheng Fan and Yu-Ching Yang and Yi-Hsuan Yang},
  title = {{Addressing The Confounds Of Accompaniments In Singer Identification}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2020},
  url = {https://openreview.net/forum?id=gCGjkNZKKc},
}

Loading related content...

Figure