Integrating End-to-End Neural and Clustering-Based Diarization: Getting the Best of Both Worlds

Keisuke Kinoshita, Marc Delcroix, Naohiro Tawara

TL;DR

This paper introduces a novel hybrid speaker diarization system, "EEND-vector clustering," that combines the strengths of end-to-end neural diarization (EEND) for handling overlapping speech and clustering-based methods for long recordings. By modifying EEND to output speaker embeddings alongside diarization labels, the system effectively resolves the inter-block permutation problem, achieving state-of-the-art performance on long, challenging conversational data.

What Problem Does This Paper Address?

Speaker diarization, the task of identifying "who spoke when," is crucial for analyzing conversations and meetings. Existing methods fall into two main categories: clustering-based approaches (like those using x-vectors) and end-to-end neural diarization (EEND). Clustering methods are robust and handle an arbitrary number of speakers but struggle with overlapping speech. EEND, on the other hand, can handle overlaps but faces significant challenges with long recordings due to memory constraints and the "inter-block label permutation problem" – where the same speaker might be assigned different labels across consecutive segments. This paper aims to bridge this gap by creating a system that excels at both handling overlapping speech and processing long recordings efficiently and accurately.

Methods

The core innovation is the "EEND-vector clustering" framework. It builds upon the EEND architecture by modifying the neural network to not only predict speaker activity for each frame but also to output a global speaker embedding for each identified speaker within a chunk. This is achieved by adding a "speaker embedding estimator" to the network.

The training objective is a multi-task loss function that balances diarization error and speaker embedding quality. The diarization loss uses permutation-invariant training to handle label ambiguities within a chunk. Crucially, the speaker embedding loss encourages embeddings from the same speaker to be close in an embedding space, while embeddings from different speakers are pushed apart.

For inference, the system segments long recordings into chunks. The modified EEND processes each chunk, producing diarization results and speaker embeddings. The inter-block permutation problem is then solved by clustering these speaker embeddings across chunks. A constrained clustering algorithm (COP-k-means) is used to ensure that embeddings from the same chunk are not assigned to the same speaker cluster, further refining the process.

Datasets

The paper uses the Librispeech database (8 kHz) for training, development, and testing. To simulate realistic conversational scenarios, they generated noisy and reverberant mixtures of two speakers. The test data was specifically designed to include varying durations (3, 5, 10, and 20 minutes) and overlap ratios (10-90%), with speakers from the Librispeech test set not seen during training.

Key Insights and Ideas

  • Bridging the Gap: The paper successfully demonstrates that combining EEND's overlap handling with clustering's long-sequence robustness is achievable and highly effective.
  • Solving Permutation Ambiguity: The novel approach of using speaker embeddings to resolve the inter-block label permutation problem is a key contribution, simplifying the stitching of diarization results across chunks.
  • Long Recordings Advantage: The proposed method shows a significant performance improvement on longer recordings, suggesting that the availability of more embeddings for clustering aids in better speaker identification.
  • Embedding Quality: The speaker embedding loss function is effective in producing discriminative embeddings, as visualized by t-SNE plots, which clearly separate different speakers even with significant overlap.

Key Results

  • The proposed "EEND-vector clustering" method significantly outperforms the conventional EEND, especially on longer test data (10 and 20 minutes).
  • For 20-minute test data, the Diarization Error Rate (DER) for the proposed method was 5.5%, compared to 10.1% for conventional EEND.
  • The method also shows superior performance across different overlap ratios, outperforming conventional EEND in all tested conditions (0-30%, 30-60%, 60-90%).
  • The addition of speaker embedding estimation did not negatively impact the diarization performance on shorter sequences, maintaining parity with the original EEND.

Limitations and Future Work

The authors acknowledge that the current framework assumes a fixed maximum number of active speakers within a chunk ($S_{Local}$). Future work will focus on extending the framework to handle an arbitrary number of speakers within a chunk, which would further enhance its applicability to real-world scenarios. They also suggest exploring more challenging conditions and potentially applying the framework to other speech processing tasks.

Summary updated

Abstract

Recent diarization technologies can be categorized into two approaches, i.e., clustering and end-to-end neural approaches, which have different pros and cons. The clustering-based approaches assign speaker labels to speech regions by clustering speaker embeddings such as x-vectors. While it can be seen as a current state-of-the-art approach that works for various challenging data with reasonable robustness and accuracy, it has a critical disadvantage that it cannot handle overlapped speech that is inevitable in natural conversational data. In contrast, the end-to-end neural diarization (EEND), which directly predicts diarization labels using a neural network, was devised to handle the overlapped speech. While the EEND, which can easily incorporate emerging deep-learning technologies, has started outperforming the x-vector clustering approach in some realistic database, it is difficult to make it work for `long' recordings (e.g., recordings longer than 10 minutes) because of, e.g., its huge memory consumption. Block-wise independent processing is also difficult because it poses an inter-block label permutation problem, i.e., an ambiguity of the speaker label assignments between blocks. In this paper, we propose a simple but effective hybrid diarization framework that works with overlapped speech and for long recordings containing an arbitrary number of speakers. It modifies the conventional EEND framework to simultaneously output global speaker embeddings so that speaker clustering can be performed across blocks to solve the permutation problem. With experiments based on simulated noisy reverberant 2-speaker meeting-like data, we show that the proposed framework works significantly better than the original EEND especially when the input data is long.
BibTeX
@inproceedings{kinoshita2021integrating,
  author = {Keisuke Kinoshita and Marc Delcroix and Naohiro Tawara},
  title = {{Integrating End-to-End Neural and Clustering-Based Diarization: Getting the Best of Both Worlds}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2021},
  doi = {10.1109/icassp39728.2021.9414333},
}
Figure