Abstract
Speaker diarization, the process of determining "who spoke when," is a fundamental task in audio processing. While deep learning has significantly improved performance through d-vector embeddings, individual model architectures often capture complementary acoustic information. This paper introduces the "c-vector," a novel speaker embedding that leverages a 2D self-attention mechanism to combine multiple neural network outputs. By weighting information across both time and model architectures, the proposed system achieves a 10% relative reduction in Speaker Error Rate (SER) on the AMI meeting corpus compared to state-of-the-art single-model systems.
Introduction
Modern speaker diarization systems rely on partitioning audio into segments represented by fixed-length embeddings. While the field has transitioned from traditional i-vectors to deep neural network (DNN) based d-vectors, no single architecture is universally optimal. Time Delay Neural Networks (TDNNs) and Recurrent Neural Networks (RNNs) exhibit complementary strengths: TDNNs excel at capturing short-term phonetic patterns, while RNNs are better suited for long-term prosodic features. The challenge lies in effectively fusing these diverse representations to create a more robust speaker identity.
Methodology
2D Self-Attentive Combination
The core innovation is the 2D self-attentive structure used to generate the combined "c-vector." Unlike standard attention that only operates across the temporal dimension, this approach extends attention across different embedding types. The researchers explored a Consecutive Architecture, a hierarchical approach that first performs temporal attention within each individual system before a secondary "master" attention layer weights the resulting outputs to form the final c-vector.
Modified Penalty Term: Smooth vs. Spiky Attention
Standard self-attention often employs a penalty term to encourage "spiky" attention, focusing on a few highly discriminative frames. However, for diarization, capturing broader acoustic trends is often beneficial. The authors introduced a tunable parameter (lambda) to the penalty term, allowing the model to produce both "spiky" (local focus) and "smooth" (overall trend) attention vectors. This diversity enables the c-vector to capture both fine-grained vocal characteristics and general environmental acoustics.
Experimental Setup
The system was evaluated using the AMI meeting corpus, a standard benchmark for multi-speaker environments. The training set comprised 135 meetings with 149 speakers. Input features consisted of 40-dimensional log mel filter bank features extracted from beam-formed audio. The study compared two distinct architectures: a TDNN (1.76M parameters) and a High Order RNN (HORNN, 0.29M parameters). Despite its significantly smaller size, the HORNN provided unique, complementary information essential for the combined system's success.
Key Results
The 2D self-attentive approach yielded substantial performance gains across the board:
- Penalty Term Impact: Simply updating the penalty term improved the standalone TDNN by 6% and the HORNN by 21% in relative SER.
- C-Vector Performance: The best 2D attentive combination (Consecutive 2) achieved an SER of 12.99% on the evaluation set.
- Comparative Advantage: This represents a 10% relative reduction over the best individual d-vector system and a massive 46% reduction compared to the DiarTK baseline (23.31% SER).
Conclusion
The introduction of c-vectors via 2D self-attention marks a significant advancement in speaker diarization. By intelligently fusing complementary DNN architectures and utilizing a flexible attention penalty term, the system captures a richer representation of speaker identity. The success of the Consecutive architecture suggests that preserving the "individuality" of each model before merging is beneficial. Future work will focus on scaling this approach to more diverse model ensembles and optimizing for real-time inference scenarios.
Summary updated