Incremental diarization of telephone conversations

Oshry Ben-Harush, Itshak Lapidot, Hugo GutermanInterspeech 2010

TL;DR

This paper introduces an "incremental" speaker diarization system for telephone conversations, allowing for real-time analysis without needing the entire conversation upfront. This is crucial for applications like forensics where immediate insights are needed from lengthy recordings. The proposed two-stage approach achieves performance close to offline methods, with a manageable increase in error.

What Problem Does This Paper Address?

Speaker diarization aims to identify "who spoke when" in an audio recording. Traditional diarization systems typically require the entire audio file to be available before processing. This "offline" approach works well for many applications, but it's a significant limitation for scenarios like forensic investigations or live monitoring, where analyzing a conversation as it unfolds is essential. The challenge is to develop a diarization system that can process audio incrementally, adapting and refining its speaker labels as new segments of the conversation become available, without sacrificing too much accuracy compared to the offline gold standard.

Methods

The authors propose a two-stage incremental diarization system:

  1. First Stage: Initial Training Set (ITS) Diarization: An initial segment of the conversation is processed using a fully unsupervised diarization algorithm. This stage is based on a previously described iterative system that uses Self-Organizing Maps (SOMs) for speaker modeling and Hidden Markov Models (HMMs) for time-series clustering. The goal here is to build initial speaker and non-speech models and tune HMM parameters.
  2. Second Stage: Incremental Diarization: The rest of the conversation is processed in small time increments. For each increment, the system uses the models and HMM parameters established in the first stage. Crucially, the speaker models are then adapted based on the new segment, and the process is repeated for subsequent increments. This adaptation step is key to maintaining performance as the conversation progresses. The paper explores the impact of the number of adaptation iterations.

The core techniques involve:

  • Self-Organizing Maps (SOMs): Used for speaker modeling, where each speaker is represented by a "Code Book" of "Code Words." This is chosen for its lower data requirement compared to traditional Gaussian Mixture Models (GMMs).
  • Hidden Markov Models (HMMs): Employed for time-series clustering to segment the conversation into speaker turns. A modified HMM with a "hyper-state transition matrix" is used to model speaker and non-speech states.
  • Weighted Segmental K-Means (WSKM): Used for the initial unsupervised segmentation and assignment of speech segments.

Datasets

The proposed system was evaluated on 1802 telephone conversations from the NIST 2005 Speaker Recognition Evaluation (SRE) dataset. These recordings are 5 minutes long, originally from a 4-wire (two-channel) setup, which were summed and normalized to create a single-channel audio stream.

Key Insights and Ideas

  • Incremental Adaptation is Key: The system's ability to adapt speaker models based on new incoming audio segments is vital for maintaining performance in an incremental setting.
  • Trade-off Between Resolution and Accuracy: Shorter incremental segments (e.g., 30 seconds) offer better temporal resolution but can lead to slightly higher diarization error compared to longer increments, especially with fewer adaptation iterations.
  • SOMs as a Viable Alternative: Using SOMs for speaker modeling proves effective, offering a simpler and less data-intensive approach than GMMs, while still achieving competitive diarization accuracy.
  • Robustness to Prior Information: The system is designed to be fully unsupervised, meaning it doesn't require any prior knowledge about the number of speakers, their identities, or environmental conditions, making it highly robust and scalable.

Key Results

  • The "optimal" diarization error rate (achieved by running the offline system on the entire conversation) was approximately 20%.
  • The incremental diarization system, using an Initial Training Set (ITS) of 2 minutes and incremental segments of 1 minute, resulted in a diarization error rate (DER) of approximately 23.9%. This is a roughly 2% increase compared to the offline system.
  • When using a 2-minute ITS and 60-second incremental segments with three learning iterations, the DER was 22.1%, which is only about 2.1% higher than the optimal offline performance.
  • The paper notes that their incremental approach is 1.9% better than a previously reported "on-line" diarization system (likely referring to a system with less adaptation or a different approach).

Limitations and Future Work

  • Overlapped Speech: The current system does not explicitly handle overlapped speech (when two speakers talk at the same time). Segments identified as overlapped speech in the reference diarization are treated as errors, contributing to the overall DER.
  • Number of Iterations: While more adaptation iterations can improve performance, especially with longer incremental segments, there's a point of diminishing returns. The paper suggests that for high-resolution diarization (short increments), fewer adaptation iterations might be preferable.
  • Future Work: The authors suggest that further exploration of the number of adaptation iterations and incremental segment lengths could lead to even better performance. The robustness and scalability of the system are highlighted as key advantages for future development.

Summary updated

BibTeX
@inproceedings{benharush2010incremental,
  author = {Oshry Ben-Harush and Itshak Lapidot and Hugo Guterman},
  title = {{Incremental diarization of telephone conversations}},
  booktitle = {Interspeech},
  year = {2010},
}
Figure