TL;DR
This paper introduces a novel hybrid speaker diarization system, "EEND-vector clustering," that combines the strengths of end-to-end neural diarization (EEND) for handling overlapping speech and clustering-based methods for long recordings. By modifying EEND to output speaker embeddings alongside diarization labels, the system effectively resolves the inter-block permutation problem, achieving state-of-the-art performance on long, challenging conversational data.
What Problem Does This Paper Address?
Speaker diarization, the task of identifying "who spoke when," is crucial for analyzing conversations and meetings. Existing methods fall into two main categories: clustering-based approaches (like those using x-vectors) and end-to-end neural diarization (EEND). Clustering methods are robust and handle an arbitrary number of speakers but struggle with overlapping speech. EEND, on the other hand, can handle overlaps but faces significant challenges with long recordings due to memory constraints and the "inter-block label permutation problem" – where the same speaker might be assigned different labels across consecutive segments. This paper aims to bridge this gap by creating a system that excels at both handling overlapping speech and processing long recordings efficiently and accurately.
Methods
The core innovation is the "EEND-vector clustering" framework. It builds upon the EEND architecture by modifying the neural network to not only predict speaker activity for each frame but also to output a global speaker embedding for each identified speaker within a chunk. This is achieved by adding a "speaker embedding estimator" to the network.
The training objective is a multi-task loss function that balances diarization error and speaker embedding quality. The diarization loss uses permutation-invariant training to handle label ambiguities within a chunk. Crucially, the speaker embedding loss encourages embeddings from the same speaker to be close in an embedding space, while embeddings from different speakers are pushed apart.
For inference, the system segments long recordings into chunks. The modified EEND processes each chunk, producing diarization results and speaker embeddings. The inter-block permutation problem is then solved by clustering these speaker embeddings across chunks. A constrained clustering algorithm (COP-k-means) is used to ensure that embeddings from the same chunk are not assigned to the same speaker cluster, further refining the process.
Datasets
The paper uses the Librispeech database (8 kHz) for training, development, and testing. To simulate realistic conversational scenarios, they generated noisy and reverberant mixtures of two speakers. The test data was specifically designed to include varying durations (3, 5, 10, and 20 minutes) and overlap ratios (10-90%), with speakers from the Librispeech test set not seen during training.
Key Insights and Ideas
- Bridging the Gap: The paper successfully demonstrates that combining EEND's overlap handling with clustering's long-sequence robustness is achievable and highly effective.
- Solving Permutation Ambiguity: The novel approach of using speaker embeddings to resolve the inter-block label permutation problem is a key contribution, simplifying the stitching of diarization results across chunks.
- Long Recordings Advantage: The proposed method shows a significant performance improvement on longer recordings, suggesting that the availability of more embeddings for clustering aids in better speaker identification.
- Embedding Quality: The speaker embedding loss function is effective in producing discriminative embeddings, as visualized by t-SNE plots, which clearly separate different speakers even with significant overlap.
Key Results
- The proposed "EEND-vector clustering" method significantly outperforms the conventional EEND, especially on longer test data (10 and 20 minutes).
- For 20-minute test data, the Diarization Error Rate (DER) for the proposed method was 5.5%, compared to 10.1% for conventional EEND.
- The method also shows superior performance across different overlap ratios, outperforming conventional EEND in all tested conditions (0-30%, 30-60%, 60-90%).
- The addition of speaker embedding estimation did not negatively impact the diarization performance on shorter sequences, maintaining parity with the original EEND.
Limitations and Future Work
The authors acknowledge that the current framework assumes a fixed maximum number of active speakers within a chunk ($S_{Local}$). Future work will focus on extending the framework to handle an arbitrary number of speakers within a chunk, which would further enhance its applicability to real-world scenarios. They also suggest exploring more challenging conditions and potentially applying the framework to other speech processing tasks.
Summary updated