Combination of Deep Speaker Embeddings for Diarisation

Guangzhi Sun, Chao Zhang, Philip C. Woodland

TL;DR

This paper introduces c-vectors, a novel approach to speaker diarisation that combines multiple complementary deep speaker embeddings (d-vectors) using sophisticated fusion techniques like 2D self-attention and bilinear pooling. By integrating these embeddings into a complete, single-pass neural diarisation pipeline, the authors achieve state-of-the-art results, including a 29% relative reduction in speaker error rate on the challenging AMI meeting corpus.

What Problem Does This Paper Address?

Speaker diarisation—the task of determining "who spoke when"—is a fundamental component of speech processing, yet it remains notoriously difficult in multi-speaker environments like business meetings. While deep learning has replaced traditional i-vectors with d-vectors (embeddings from neural networks), individual model architectures often have specific strengths and weaknesses.

A single model might excel at capturing temporal patterns but struggle with spectral noise, or vice versa. The authors address the challenge of how to effectively combine these diverse, complementary embeddings to create a more robust and accurate representation of speaker identity, especially in "realistic" settings involving multiple distant microphones and 4–10 speakers.

Methods

The authors propose a comprehensive neural diarisation pipeline and several novel embedding combination structures.

The Neural Pipeline

The system consists of four main stages:

  1. Neural VAD: A deep neural network for Voice Activity Detection.
  2. Neural CPD: A novel Change Point Detection model that uses a Recurrent Neural Network (RNN) to compare speaker representations before and after a potential change point. Crucially, it uses pre-trained d-vectors as inputs rather than raw features.
  3. Embedding Extraction & Combination: This is the core contribution. Multiple d-vectors (e.g., from TDNN and HORNN architectures) are fused into a c-vector.
  4. Spectral Clustering: Grouping the c-vectors to assign speaker labels.
Figure 1: Illustration of the speaker diarisation system. Path (a) denotes the window-level clustering pipeline. Path (b) denotes the full pipeline with the CPD component, which leads to the speaker-homogeneous segment-level clustering results.

C-Vector Combination Structures

The paper explores three primary fusion methods:

  • 2D Self-Attentive Combination: Extends standard attention to two dimensions—integrating frame-level features across time and fusing window-level embeddings across different systems.
  • Gated Additive Combination: Uses a gating mechanism to dynamically scale individual elements of each embedding before summing them, allowing for flexible, element-wise fusion.
  • Bilinear Pooling: Captures multiplicative interactions between every pair of elements from two different embeddings. The authors use a modified low-rank approximation with residual connections to keep the parameter count manageable while improving stability.

The best performance was achieved by a Stacked Combination, which first applies 2D self-attention and then fuses the result with another embedding via bilinear pooling.

Figure 4: A sketch map of the 2D self-attentive combination applied to 퐾window-level d-vector extraction systems.

Datasets

  • AMI Meeting Corpus: The primary benchmark, using the Multiple Distance Microphone (MDM) setup. This is considered more realistic and difficult than the headset microphone setups often used in other studies.
  • NIST RT05: Used as an additional evaluation set to demonstrate the system's robustness across different acoustic conditions and speaker counts (up to 10).
  • VoxCeleb 1 & 2: Large-scale speaker recognition datasets used for pre-training the embedding extractors to improve generalisation.

Key Insights and Ideas

  • Architecture Complementarity: The authors show that Time-Delay Neural Networks (TDNNs) and High-Order Recurrent Neural Networks (HORNNs) capture different speaker characteristics. Combining them yields better results than scaling up either individual architecture.
  • Multiplicative Interaction: Bilinear pooling's ability to model multiplicative interactions between features proves superior to simple additive fusion, as it provides a richer representation of the speaker space.
  • Neural CPD with Embeddings: Using d-vectors as input to the Change Point Detector (Figure 2) significantly outperforms traditional distance-based methods. It allows the model to make decisions based on high-level speaker identity rather than low-level acoustic shifts.
Proposed neural CPD structure using pre-trained TDNNs and RNNs to detect speaker changes.

Key Results

  • SER Reductions: On the AMI evaluation set, the c-vector approach achieved a 29% relative reduction in Speaker Error Rate (SER) compared to the best individual d-vector system.
  • Cross-Task Robustness: On the NIST RT05 set, the system achieved a 15% relative SER reduction, confirming that the improvements generalize beyond the AMI corpus.
  • Large-Scale Training: Incorporating VoxCeleb data further lowered the Diarisation Error Rate (DER) across all sets, with the best combined system reaching a DER of 13.8% on AMI eval (without scoring overlap).

Limitations and Future Work

  • Clustering Integration: While the pipeline is mostly neural, the final clustering step (Spectral Clustering) remains a separate, non-neural algorithm. The authors suggest that moving toward a fully end-to-end neural clustering approach is a logical next step.
  • Overlapping Speech: The current scoring excludes overlapping speech regions. Handling "who spoke when" during simultaneous talk remains a major open challenge in the field.
  • Parameter Efficiency: While low-rank bilinear pooling helps, these combination structures increase the total parameter count. Future work could explore even more efficient fusion methods for resource-constrained environments.

Summary updated

Abstract

Significant progress has recently been made in speaker diarisation after the introduction of d-vectors as speaker embeddings extracted from neural network (NN) speaker classifiers for clustering speech segments. To extract better-performing and more robust speaker embeddings, this paper proposes a c-vector method by combining multiple sets of complementary d-vectors derived from systems with different NN components. Three structures are used to implement the c-vectors, namely 2D self-attentive, gated additive, and bilinear pooling structures, relying on attention mechanisms, a gating mechanism, and a low-rank bilinear pooling mechanism respectively. Furthermore, a neural-based single-pass speaker diarisation pipeline is also proposed in this paper, which uses NNs to achieve voice activity detection, speaker change point detection, and speaker embedding extraction. Experiments and detailed analyses are conducted on the challenging AMI and NIST RT05 datasets which consist of real meetings with 4--10 speakers and a wide range of acoustic conditions. For systems trained on the AMI training set, relative speaker error rate (SER) reductions of 13% and 29% are obtained by using c-vectors instead of d-vectors on the AMI dev and eval sets respectively, and a relative reduction of 15% in SER is observed on RT05, which shows the robustness of the proposed methods. By incorporating VoxCeleb data into the training set, the best c-vector system achieved 7%, 17% and16% relative SER reduction compared to the d-vector on the AMI dev, eval, and RT05 sets respectively
BibTeX
@article{sun2021combination,
  author = {Guangzhi Sun and Chao Zhang and Philip C. Woodland},
  title = {{Combination of Deep Speaker Embeddings for Diarisation}},
  journal = {Neural Networks},
  year = {2021},
  doi = {10.1016/j.neunet.2021.04.020},
}
Figure