Speaker Diarisation Using 2D Self-attentive Combination of Embeddings

Guangzhi Sun, Chao Zhang, Philip C. Woodland

Abstract

Speaker diarization, the process of determining "who spoke when," is a fundamental task in audio processing. While deep learning has significantly improved performance through d-vector embeddings, individual model architectures often capture complementary acoustic information. This paper introduces the "c-vector," a novel speaker embedding that leverages a 2D self-attention mechanism to combine multiple neural network outputs. By weighting information across both time and model architectures, the proposed system achieves a 10% relative reduction in Speaker Error Rate (SER) on the AMI meeting corpus compared to state-of-the-art single-model systems.

Introduction

Modern speaker diarization systems rely on partitioning audio into segments represented by fixed-length embeddings. While the field has transitioned from traditional i-vectors to deep neural network (DNN) based d-vectors, no single architecture is universally optimal. Time Delay Neural Networks (TDNNs) and Recurrent Neural Networks (RNNs) exhibit complementary strengths: TDNNs excel at capturing short-term phonetic patterns, while RNNs are better suited for long-term prosodic features. The challenge lies in effectively fusing these diverse representations to create a more robust speaker identity.

Methodology

2D Self-Attentive Combination

The core innovation is the 2D self-attentive structure used to generate the combined "c-vector." Unlike standard attention that only operates across the temporal dimension, this approach extends attention across different embedding types. The researchers explored a Consecutive Architecture, a hierarchical approach that first performs temporal attention within each individual system before a secondary "master" attention layer weights the resulting outputs to form the final c-vector.

The Consecutive Architecture hierarchical approach to embedding fusion.

Modified Penalty Term: Smooth vs. Spiky Attention

Standard self-attention often employs a penalty term to encourage "spiky" attention, focusing on a few highly discriminative frames. However, for diarization, capturing broader acoustic trends is often beneficial. The authors introduced a tunable parameter (lambda) to the penalty term, allowing the model to produce both "spiky" (local focus) and "smooth" (overall trend) attention vectors. This diversity enables the c-vector to capture both fine-grained vocal characteristics and general environmental acoustics.

Visualization of spiky vs. smooth attention vectors controlled by the penalty term parameter lambda.

Experimental Setup

The system was evaluated using the AMI meeting corpus, a standard benchmark for multi-speaker environments. The training set comprised 135 meetings with 149 speakers. Input features consisted of 40-dimensional log mel filter bank features extracted from beam-formed audio. The study compared two distinct architectures: a TDNN (1.76M parameters) and a High Order RNN (HORNN, 0.29M parameters). Despite its significantly smaller size, the HORNN provided unique, complementary information essential for the combined system's success.

Key Results

The 2D self-attentive approach yielded substantial performance gains across the board:

  • Penalty Term Impact: Simply updating the penalty term improved the standalone TDNN by 6% and the HORNN by 21% in relative SER.
  • C-Vector Performance: The best 2D attentive combination (Consecutive 2) achieved an SER of 12.99% on the evaluation set.
  • Comparative Advantage: This represents a 10% relative reduction over the best individual d-vector system and a massive 46% reduction compared to the DiarTK baseline (23.31% SER).

Conclusion

The introduction of c-vectors via 2D self-attention marks a significant advancement in speaker diarization. By intelligently fusing complementary DNN architectures and utilizing a flexible attention penalty term, the system captures a richer representation of speaker identity. The success of the Consecutive architecture suggests that preserving the "individuality" of each model before merging is beneficial. Future work will focus on scaling this approach to more diverse model ensembles and optimizing for real-time inference scenarios.

Summary updated

Abstract

Speaker diarisation systems often cluster audio segments using speaker embeddings such as i-vectors and d-vectors. Since different types of embeddings are often complementary, this paper proposes a generic framework to improve performance by combining them into a single embedding, referred to as a c-vector. This combination uses a 2-dimensional (2D) self-attentive structure, which extends the standard self-attentive layer by averaging not only across time but also across different types of embeddings. Two types of 2D self-attentive structure in this paper are the simultaneous combination and the consecutive combination, adopting a single and multiple self-attentive layers respectively. The penalty term in the original self-attentive layer which is jointly minimised with the objective function to encourage diversity of annotation vectors is also modified to obtain not only different local peaks but also the overall trends in the multiple annotation vectors. Experiments on the AMI meeting corpus show that our modified penalty term improves the d- vector relative speaker error rate (SER) by 6% and 21% for d-vector systems, and a 10% further relative SER reduction can be obtained using the c-vector from our best 2D self-attentive structure.
BibTeX
@inproceedings{sun2019speaker,
  author = {Guangzhi Sun and Chao Zhang and Philip C. Woodland},
  title = {{Speaker Diarisation Using 2D Self-attentive Combination of Embeddings}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2019},
  doi = {10.1109/icassp.2019.8683373},
}
Figure