Introduction
Speaker diarization, the process of partitioning an audio stream into homogeneous segments according to speaker identity, is a fundamental task in automatic speech processing. While significant progress has been made in single-speaker scenarios, performance degrades sharply in the presence of overlapped speech. This paper presents a robust overlap detection system that fuses traditional spectral features with spatial information derived from microphone arrays, achieving substantial improvements in diarization accuracy.
The Challenge of Overlapped Speech
Conventional diarization systems are typically "monophonic," assuming only one active speaker at any time. This assumption leads to two primary failure modes when overlaps occur:
- Missed Speech: The system assigns only one label, ignoring secondary speakers and increasing the missed speech error.
- Model Corruption: Overlapped segments often "pollute" the statistical models used for individual speakers. Training on these segments results in less discriminative models, propagating errors throughout the entire recording.
Detecting these overlaps is particularly challenging in "distant-channel" scenarios, such as meetings recorded with far-field microphones, where reverberation and background noise further obscure the signal.
Proposed Methodology
The proposed system enhances overlap detection by integrating two distinct feature streams: spectral and spatial.
Feature Extraction and Fusion
The system utilizes standard Spectral Features (MFCCs, LPC residual energy, and spectral flatness) to capture acoustic characteristics. To leverage the multi-channel setup, the authors introduce Spatial Features based on the Generalized Cross Correlation with Phase Transform (GCC-PHAT). These include:
- Coherence Value: Indicates the strength of the correlation peak, which typically drops during overlaps.
- Coherence Dispersion Ratio: Measures the presence of secondary acoustic sources.
- Delta of Time Delay Estimation (TDE): Captures the rapid "jitter" in spatial estimates that occurs when the dominant source flickers between two speakers.
To handle the high dimensionality of data from large microphone arrays (e.g., 66 pairs for a 12-mic array), two fusion strategies were evaluated: Principal Component Analysis (PCA) for linear dimensionality reduction and Artificial Neural Networks (ANN) for non-linear classification scoring.
Integration into Diarization
The detected overlap hypotheses are integrated into the diarization pipeline through two mechanisms:
- Exclusion: Removing overlap segments from speaker model training to ensure high-quality, "pure" models.
- Labeling: Assigning a second speaker label during the final pass to recover speech that would otherwise be missed.
Experimental Evaluation
The system was evaluated on the AMI Meeting Corpus, using 100 hours of multi-party conversation. The researchers tested the system's robustness across two scenarios:
- Single-Site: Recordings from a consistent acoustic environment.
- Multi-Site: Recordings from diverse rooms with varying geometries and microphone configurations.
A key finding was the necessity of Tailored Hypotheses. The "Exclusion" task benefits from high precision to avoid discarding valid training data, whereas "Labeling" requires high recall to capture as much overlapped speech as possible.
Key Results
The fusion of spatial and spectral features consistently outperformed the spectral-only baseline. In multi-site evaluations, the ANN-based fusion proved particularly robust, effectively generalizing across different acoustic environments.
As shown in the results, the combination of exclusion and labeling ("+Both") yielded the most significant gains. For multi-site data, the ANN approach achieved a 14.6% relative improvement in Diarization Error Rate (DER), reducing it from a baseline of 37.3% to 31.9%. In single-site scenarios, the PCA method achieved an 11.6% relative improvement.
Conclusion
This research demonstrates that spatial information is a powerful cue for detecting overlapped speech in distant-microphone settings. By effectively fusing these features with traditional spectral data, diarization systems can significantly mitigate the impact of overlaps. Future work will focus on reducing the computational cost of multi-pair cross-correlation and developing dynamic weighting schemes to adapt to varying signal-to-noise ratios in real-time.
Summary updated