Speaker segmentation and clustering for simultaneously presented speech

Lingyun Gu, Richard M. SternInterspeech 2009

TL;DR

This paper introduces a novel method for segmenting and clustering speech from multiple speakers who are talking at the same time, even when their voices are heavily overlapped and at different loudness levels. The key innovation lies in new features derived from the time-frequency representation of speech and a two-stage segmentation process, leading to improved performance over existing methods with minimal computational overhead.

What Problem Does This Paper Address?

Identifying and separating individual speakers from a mixed audio stream is a crucial task in many applications, from improving speaker recognition systems to enabling better audio indexing for broadcast content. The challenge intensifies significantly when multiple speakers are talking simultaneously, especially when their speech overlaps heavily and their signal strengths (Signal-to-Noise Ratio or SNR) vary. Traditional methods often struggle with short speech segments and highly overlapping speech, and their performance degrades in noisy or complex acoustic environments. This paper specifically tackles the problem of segmenting and clustering speech from two simultaneously presented speakers, even when their utterances are as short as 2-3 seconds and almost completely overlapping.

Methods

The proposed system employs a two-stage approach. First, it decomposes the incoming speech into a two-dimensional time-frequency representation using Short-Time Fourier Analysis (STFA). A novel feature extraction method is introduced: it focuses on the strongest time-frequency cells (top 20% in power) and derives two features: the count of these high-energy cells per frame and the ratio of high-frequency to low-frequency energy. These are complemented by conventional features like pitch, kurtosis, and zero-crossing rate.

The segmentation process is a two-stage filtering procedure. The first stage uses the novel energy-based features to identify potential speaker-change points by looking for valleys in the "energy" curve and dramatic changes in the high/low frequency ratio. The second stage combines pitch, kurtosis, and zero-crossing rate features, calculating the Euclidean distance between consecutive frames to detect further change points. A pool of potential speaker-change candidates is generated by merging results from both stages.

For clustering, the paper uses a distance measure between Gaussian distributions, specifically a simplified version of the divergence between multivariate Gaussian probability density functions. Segments with small distances are grouped together, assuming two speakers a priori.

Datasets

The authors created a custom dataset for evaluation. They digitally added 1600 pairs of standard DARPA Resource Management (RM) sentences at SNRs of 5 dB, 10 dB, and 15 dB. For each SNR, they generated four combinations: male-male, female-female, male-female, and female-male. This dataset was designed to specifically address the lack of simultaneously presented speech segments in existing benchmarks.

Key Insights and Ideas

  • Novel Features for Overlap: The core idea is that focusing on the most prominent time-frequency components (highest energy cells) can reveal underlying speech structures even in highly mixed signals. This is a departure from traditional methods that might use all components or rely solely on standard acoustic features.
  • Two-Stage Segmentation for Robustness: Combining features derived from time-frequency analysis with traditional time-domain features in a two-stage process helps to overcome the limitations of each individual feature set and improve the robustness of change-point detection.
  • Gender Matters: The results highlight that separating speech is significantly easier when speakers have different genders, likely due to distinct pitch contours and harmonic structures. This is an important consideration for real-world applications.
  • Segmentation Errors Don't Always Kill Clustering: Interestingly, the paper notes that segmentation errors don't always lead to poor clustering performance. This can happen when errors occur during frames where speaker energies are very close, which are often excluded from analysis anyway.

Key Results

The proposed system consistently outperformed a baseline system (using MFCC features, BIC for change-point detection, and KL distance for clustering) across various SNRs.

  • Segmentation Performance: The new system achieved higher F-measures compared to the conventional system at all tested SNRs (e.g., 0.72 vs. 0.69 at 5 dB, 0.76 vs. 0.73 at 10 dB, and 0.85 vs. 0.80 at 15 dB).
  • Clustering Performance: Similarly, clustering performance was also superior, with F-measures of 0.82 vs. 0.70 at 5 dB, 0.86 vs. 0.76 at 10 dB, and 0.80 vs. 0.79 at 15 dB.
  • Computational Cost: The paper emphasizes that these improvements were achieved with "very small computational cost."

Limitations and Future Work

The authors acknowledge that the new features might not perfectly match clean environments, potentially explaining why the performance doesn't increase linearly with SNR. They also note that while the system performs well, segmentation errors can still occur, particularly during frames where target and masker speech have very similar energy levels. Future work could explore more sophisticated methods for handling these challenging frames and potentially extend the system to handle more than two simultaneous speakers.

Summary updated

BibTeX
@inproceedings{gu2009speaker,
  author = {Lingyun Gu and Richard M. Stern},
  title = {{Speaker segmentation and clustering for simultaneously presented speech}},
  booktitle = {Interspeech},
  year = {2009},
}
Figure