TL;DR
This paper introduces c-vectors, a novel approach to speaker diarisation that combines multiple complementary deep speaker embeddings (d-vectors) using sophisticated fusion techniques like 2D self-attention and bilinear pooling. By integrating these embeddings into a complete, single-pass neural diarisation pipeline, the authors achieve state-of-the-art results, including a 29% relative reduction in speaker error rate on the challenging AMI meeting corpus.
What Problem Does This Paper Address?
Speaker diarisation—the task of determining "who spoke when"—is a fundamental component of speech processing, yet it remains notoriously difficult in multi-speaker environments like business meetings. While deep learning has replaced traditional i-vectors with d-vectors (embeddings from neural networks), individual model architectures often have specific strengths and weaknesses.
A single model might excel at capturing temporal patterns but struggle with spectral noise, or vice versa. The authors address the challenge of how to effectively combine these diverse, complementary embeddings to create a more robust and accurate representation of speaker identity, especially in "realistic" settings involving multiple distant microphones and 4–10 speakers.
Methods
The authors propose a comprehensive neural diarisation pipeline and several novel embedding combination structures.
The Neural Pipeline
The system consists of four main stages:
- Neural VAD: A deep neural network for Voice Activity Detection.
- Neural CPD: A novel Change Point Detection model that uses a Recurrent Neural Network (RNN) to compare speaker representations before and after a potential change point. Crucially, it uses pre-trained d-vectors as inputs rather than raw features.
- Embedding Extraction & Combination: This is the core contribution. Multiple d-vectors (e.g., from TDNN and HORNN architectures) are fused into a c-vector.
- Spectral Clustering: Grouping the c-vectors to assign speaker labels.
C-Vector Combination Structures
The paper explores three primary fusion methods:
- 2D Self-Attentive Combination: Extends standard attention to two dimensions—integrating frame-level features across time and fusing window-level embeddings across different systems.
- Gated Additive Combination: Uses a gating mechanism to dynamically scale individual elements of each embedding before summing them, allowing for flexible, element-wise fusion.
- Bilinear Pooling: Captures multiplicative interactions between every pair of elements from two different embeddings. The authors use a modified low-rank approximation with residual connections to keep the parameter count manageable while improving stability.
The best performance was achieved by a Stacked Combination, which first applies 2D self-attention and then fuses the result with another embedding via bilinear pooling.
Datasets
- AMI Meeting Corpus: The primary benchmark, using the Multiple Distance Microphone (MDM) setup. This is considered more realistic and difficult than the headset microphone setups often used in other studies.
- NIST RT05: Used as an additional evaluation set to demonstrate the system's robustness across different acoustic conditions and speaker counts (up to 10).
- VoxCeleb 1 & 2: Large-scale speaker recognition datasets used for pre-training the embedding extractors to improve generalisation.
Key Insights and Ideas
- Architecture Complementarity: The authors show that Time-Delay Neural Networks (TDNNs) and High-Order Recurrent Neural Networks (HORNNs) capture different speaker characteristics. Combining them yields better results than scaling up either individual architecture.
- Multiplicative Interaction: Bilinear pooling's ability to model multiplicative interactions between features proves superior to simple additive fusion, as it provides a richer representation of the speaker space.
- Neural CPD with Embeddings: Using d-vectors as input to the Change Point Detector (Figure 2) significantly outperforms traditional distance-based methods. It allows the model to make decisions based on high-level speaker identity rather than low-level acoustic shifts.
Key Results
- SER Reductions: On the AMI evaluation set, the c-vector approach achieved a 29% relative reduction in Speaker Error Rate (SER) compared to the best individual d-vector system.
- Cross-Task Robustness: On the NIST RT05 set, the system achieved a 15% relative SER reduction, confirming that the improvements generalize beyond the AMI corpus.
- Large-Scale Training: Incorporating VoxCeleb data further lowered the Diarisation Error Rate (DER) across all sets, with the best combined system reaching a DER of 13.8% on AMI eval (without scoring overlap).
Limitations and Future Work
- Clustering Integration: While the pipeline is mostly neural, the final clustering step (Spectral Clustering) remains a separate, non-neural algorithm. The authors suggest that moving toward a fully end-to-end neural clustering approach is a logical next step.
- Overlapping Speech: The current scoring excludes overlapping speech regions. Handling "who spoke when" during simultaneous talk remains a major open challenge in the field.
- Parameter Efficiency: While low-rank bilinear pooling helps, these combination structures increase the total parameter count. Future work could explore even more efficient fusion methods for resource-constrained environments.
Summary updated