Speaker clustering for a mixture of singing and reading

Mahnoosh Mehrabani, John H. L. HansenInterspeech 2012

TL;DR

This paper tackles the challenge of speaker clustering when speech samples include diverse vocal styles like singing and reading. They propose a two-stage approach that first clusters speech segments using traditional methods and then refines these clusters in a lower-dimensional subspace learned by Probabilistic Linear Discriminant Analysis (PLDA), significantly improving accuracy for mixed-style speech.

What Problem Does This Paper Address?

Speaker clustering is crucial for organizing large audio archives, enabling tasks like audio indexing and retrieval. However, existing systems often struggle when speakers exhibit variations in their vocal production, such as changes in speaking style, emotional state, or even singing. Singing, in particular, drastically alters the time-frequency structure of a voice, making it difficult to distinguish speakers. This paper aims to improve speaker clustering robustness by specifically addressing the challenge of differentiating speakers when their speech samples include both normal reading and singing. This has direct implications for music information retrieval and general audio analysis where diverse vocal styles are common.

Methods

The authors propose a two-stage speaker clustering algorithm:

  1. Baseline Clustering: Initial clusters are formed using traditional methods like k-means or hierarchical clustering. These methods operate on GMM (Gaussian Mixture Model) mean supervectors, which are high-dimensional representations of acoustic features extracted from speech segments.
  2. PLDA Subspace Refinement: The initial clusters are then refined within a lower-dimensional subspace learned by Probabilistic Linear Discriminant Analysis (PLDA). PLDA is a generative model that aims to maximize between-speaker variability while minimizing within-speaker variability, making it robust to variations like speaking style. The refinement process uses a log-likelihood ratio (LLR) calculated based on the trained PLDA model to decide if a speech sample should be moved to a different cluster. This iterative refinement process aims to correct misclassifications from the initial stage.

Datasets

  • UT-Sing Corpus: A newly collected corpus specifically for this study. It contains reading and singing speech samples from 33 English-speaking subjects (18 female, 15 male). Each speaker provided 5 songs for singing and read the lyrics of those songs. The singing was recorded without background music.
  • UT-VocalEffort Corpus: Used to model vocal effort classification. It consists of speech from 12 native English male speakers, each reading 20 TIMIT sentences with five vocal efforts (whispered, soft, neutral, loud, shouted).
  • TIMIT Corpus: Used to train the speaker-independent Universal Background Model (UBM) for the baseline system, as UT-Sing did not have sufficient data for this purpose.

Key Insights and Ideas

  • Singing as a Major Challenge: The study highlights that singing introduces significant changes in speech production, leading to a substantial drop in speaker clustering accuracy compared to reading alone.
  • PLDA for Style Normalization: PLDA proves effective in creating a speaker-dependent subspace that is less sensitive to speaking style variations, allowing for more accurate clustering even with mixed styles.
  • Two-Stage Approach Benefits: Combining a traditional clustering method with a PLDA-based refinement stage yields better results than either method alone, especially for mixed-style data.
  • Iterative Refinement is Key: Repeating the PLDA refinement process multiple times improves clustering accuracy, particularly for the challenging mixed-style scenario.

Key Results

The proposed two-stage algorithm, combining k-means with PLDA refinement, achieved the best results for the mixture of reading and singing speech. Specifically:

  • The baseline k-means clustering achieved an accuracy of 80.6% for the mixture.
  • The proposed k-means + PLDA refinement algorithm achieved 89.9% accuracy for the mixture after 3 refining iterations.
  • For reading-only data, accuracies remained very high (around 99%), and for singing-only data, they were also strong (around 92%), with the PLDA refinement showing slight improvements or maintaining baseline performance.
  • The study demonstrated that introducing singing can decrease clustering accuracy by up to 20% compared to reading alone.

Limitations and Future Work

  • Limitations:
    • The study assumes the number of speaker clusters is known beforehand.
  • The experiments focused on the English portion of the UT-Sing corpus.
  • The baseline system's UBM was trained on TIMIT, which might introduce some domain mismatch.
  • Future Work:
    • Applying the algorithm to speaker clustering tasks with even more speaking styles.
  • Exploring acoustic features that are inherently less sensitive to speaking style variations.
  • Extending the algorithm to handle an unknown number of speakers.

Summary updated

BibTeX
@inproceedings{mehrabani2012speaker,
  author = {Mahnoosh Mehrabani and John H. L. Hansen},
  title = {{Speaker clustering for a mixture of singing and reading}},
  booktitle = {Interspeech},
  year = {2012},
}
Figure