TL;DR
This paper introduces a novel two-pass system for speaker diarization that specifically tackles the challenge of overlapping speech. By automatically detecting and modeling overlapping speech, the system significantly improves diarization accuracy, particularly in challenging meeting environments.
What Problem Does This Paper Address?
Speaker diarization, the task of identifying "who spoke when" in an audio recording, is crucial for many applications like meeting summarization and transcription. However, a major hurdle in achieving high accuracy is the presence of overlapping speech, where multiple people talk simultaneously. Traditional diarization systems struggle with this because they typically assign each audio segment to a single speaker, leading to incorrect assignments for overlapping speech. Furthermore, overlapping speech acts as "noise" during the training of speaker models, degrading their precision and thus the overall diarization performance. This paper aims to mitigate these issues by developing a robust method for detecting and handling overlapping speech within a diarization system.
Methods
The core innovation lies in a two-pass approach.
- Overlap Detection Model Generation: In the first pass, the system automatically generates a model for overlapping speech. This model is trained on segments immediately before and after speaker changes, based on the assumption that these are high-probability overlap regions. This approach avoids the need for pre-recorded training data for overlap detection.
- Two-Pass Diarization:
- Pass 1 (Overlap Handling during Training): The generated overlap model is incorporated into the Hidden Markov Model (HMM) used for speaker diarization. This allows the system to assign overlapping speech segments to the "overlap model" rather than to individual speaker models during the training phase. This results in "purer" speaker models, less contaminated by overlapping speech.
- Pass 2 (Overlap Handling during Segmentation & Selection): The overlap model is used again in the final Viterbi segmentation. Additionally, a selection mechanism is employed to decide whether to use the overlap detection results for the final output. This selection is based on comparing the total likelihoods of the system run with and without overlap detection, ensuring that overlap detection is only applied when it demonstrably improves performance. A simple heuristic is used to assign detected overlap segments to the bordering speakers.
The system also incorporates a delay feature stream, derived from beamforming, which has previously shown to improve diarization performance.
Datasets
The experiments were conducted on a test set of 27 conference meetings from earlier NIST Rich Transcription (RT) evaluations. The paper also refers to a development set for evaluating the baseline system.
Key Insights and Ideas
- Automatic Overlap Detection is Key: The paper demonstrates that automatically detecting and modeling overlapping speech, without requiring pre-annotated training data, is a viable and effective strategy for improving diarization.
- Two-Pass System for Robustness: A two-pass approach is crucial. The first pass cleans up speaker model training by isolating overlap, while the second pass leverages this for better segmentation and includes a selection mechanism to avoid degrading performance when overlap detection isn't beneficial.
- Overlap as "Noise" During Training: A significant insight is that overlapping speech acts as detrimental noise during speaker model training. By explicitly modeling and separating it, speaker models become more accurate.
- Delay Features and Overlap Interaction: The interaction between delay features and overlap detection is complex. While delay features improve the baseline, their benefit with overlap detection is less pronounced in the full MDM system, suggesting potential avenues for future research.
Key Results
The proposed two-pass overlap detection system consistently improved diarization performance across three different system configurations (SDM, MDM without delay features, and MDM with delay features).
- Overall Improvement: The overlap detection method improved the overall system performance by 0.8%, 0.73%, and 0.07% absolute DER for the respective systems.
- MDM with Delay Features: The full MDM system (with delay features) achieved the lowest baseline DER of 12.90%. While the overlap detection provided a small absolute improvement (0.07%), it was only selected for 6 out of 27 meetings, indicating its selective benefit.
- Heuristic Assignment: The straightforward heuristic for assigning overlap segments to bordering speakers was found to be successful, although it led to an increase in false alarms, suggesting room for improvement.
Limitations and Future Work
- Heuristic Assignment Limitations: The current method for assigning overlap segments assumes overlap is always between two speakers and is biased towards interruptions. This can lead to false alarms. Future work will explore more complex models for overlap detection and assignment, potentially leveraging delay features more effectively.
- Delay Features and Overlap: The interaction between delay features and overlap detection needs further investigation. The authors hypothesize that delay features might vary significantly with overlap direction, making a single Gaussian model insufficient. Future work may involve using multiple Gaussians for delay features in overlap detection.
- Selective Application: While the system can automatically select when to apply overlap detection, the gain for the full MDM system was limited. Further research is needed to maximize the benefit of overlap detection, especially when combined with advanced features like delay streams.
Summary updated