Overlap detection for speaker diarization by fusing spectral and spatial features

Martin Zelenák, Carlos Segura, Javier HernandoInterspeech 2010 · DOI

Introduction

Speaker diarization, the process of partitioning an audio stream into homogeneous segments according to speaker identity, is a fundamental task in automatic speech processing. While significant progress has been made in single-speaker scenarios, performance degrades sharply in the presence of overlapped speech. This paper presents a robust overlap detection system that fuses traditional spectral features with spatial information derived from microphone arrays, achieving substantial improvements in diarization accuracy.

The Challenge of Overlapped Speech

Conventional diarization systems are typically "monophonic," assuming only one active speaker at any time. This assumption leads to two primary failure modes when overlaps occur:

  1. Missed Speech: The system assigns only one label, ignoring secondary speakers and increasing the missed speech error.
  2. Model Corruption: Overlapped segments often "pollute" the statistical models used for individual speakers. Training on these segments results in less discriminative models, propagating errors throughout the entire recording.

Detecting these overlaps is particularly challenging in "distant-channel" scenarios, such as meetings recorded with far-field microphones, where reverberation and background noise further obscure the signal.

Proposed Methodology

The proposed system enhances overlap detection by integrating two distinct feature streams: spectral and spatial.

Feature Extraction and Fusion

The system utilizes standard Spectral Features (MFCCs, LPC residual energy, and spectral flatness) to capture acoustic characteristics. To leverage the multi-channel setup, the authors introduce Spatial Features based on the Generalized Cross Correlation with Phase Transform (GCC-PHAT). These include:

  • Coherence Value: Indicates the strength of the correlation peak, which typically drops during overlaps.
  • Coherence Dispersion Ratio: Measures the presence of secondary acoustic sources.
  • Delta of Time Delay Estimation (TDE): Captures the rapid "jitter" in spatial estimates that occurs when the dominant source flickers between two speakers.
System architecture showing the fusion of spectral and spatial features for overlap detection.

To handle the high dimensionality of data from large microphone arrays (e.g., 66 pairs for a 12-mic array), two fusion strategies were evaluated: Principal Component Analysis (PCA) for linear dimensionality reduction and Artificial Neural Networks (ANN) for non-linear classification scoring.

Integration into Diarization

The detected overlap hypotheses are integrated into the diarization pipeline through two mechanisms:

  • Exclusion: Removing overlap segments from speaker model training to ensure high-quality, "pure" models.
  • Labeling: Assigning a second speaker label during the final pass to recover speech that would otherwise be missed.

Experimental Evaluation

The system was evaluated on the AMI Meeting Corpus, using 100 hours of multi-party conversation. The researchers tested the system's robustness across two scenarios:

  • Single-Site: Recordings from a consistent acoustic environment.
  • Multi-Site: Recordings from diverse rooms with varying geometries and microphone configurations.

A key finding was the necessity of Tailored Hypotheses. The "Exclusion" task benefits from high precision to avoid discarding valid training data, whereas "Labeling" requires high recall to capture as much overlapped speech as possible.

Key Results

The fusion of spatial and spectral features consistently outperformed the spectral-only baseline. In multi-site evaluations, the ANN-based fusion proved particularly robust, effectively generalizing across different acoustic environments.

Diarization Error Rate (DER) improvements using different overlap detection strategies on multi-site data.

As shown in the results, the combination of exclusion and labeling ("+Both") yielded the most significant gains. For multi-site data, the ANN approach achieved a 14.6% relative improvement in Diarization Error Rate (DER), reducing it from a baseline of 37.3% to 31.9%. In single-site scenarios, the PCA method achieved an 11.6% relative improvement.

Conclusion

This research demonstrates that spatial information is a powerful cue for detecting overlapped speech in distant-microphone settings. By effectively fusing these features with traditional spectral data, diarization systems can significantly mitigate the impact of overlaps. Future work will focus on reducing the computational cost of multi-pair cross-correlation and developing dynamic weighting schemes to adapt to varying signal-to-noise ratios in real-time.

Summary updated

BibTeX
@inproceedings{zelenak2010overlap,
  author = {Martin Zelenák and Carlos Segura and Javier Hernando},
  title = {{Overlap detection for speaker diarization by fusing spectral and spatial features}},
  booktitle = {Interspeech},
  year = {2010},
  doi = {10.21437/interspeech.2010-631},
}
Figure