TL;DR
This paper introduces a novel training strategy for Transformer-based End-to-End Neural Speaker Diarization (EEND) models by assigning auxiliary losses to specific self-attention heads. By identifying redundant "identity-like" attention heads and forcing them to learn speaker-wise voice activity and overlap detection patterns, the authors significantly reduce diarization error rates on both simulated and real conversational datasets.
What Problem Does This Paper Address?
Speaker diarization—solving the question of "who spoke when"—is critical for processing multi-speaker audio like meetings, interviews, and phone calls. Traditional systems use a pipeline of independent modules (voice activity detection, clustering, etc.), which are prone to compounding errors. End-to-End Neural Diarization (EEND) models solve this by directly predicting speaker activities using multi-head self-attention (MHSA) mechanisms.
However, standard self-attentive EEND (SA-EEND) models suffer from structural redundancy. As the network gets deeper, many attention heads default to "identity-like" patterns, meaning they primarily attend to the frame itself rather than capturing broader temporal or contextual relationships. This waste of model capacity limits performance, particularly in challenging scenarios like overlapping speech.
Methods
To combat this redundancy, the authors propose injecting auxiliary tasks directly into the underutilized attention heads during training.
The process follows three main steps:
- Identifying Redundant Heads: The authors calculate the mathematical trace of the attention weight matrices for each head. A high trace indicates that the head behaves like an identity matrix (focusing only on the diagonal). The heads with the highest traces are selected for regularization.
- Speaker-wise Voice Activity Detection (SVAD) Loss: Selected heads are forced to match a target mask derived from true individual speaker labels using Binary Cross-Entropy (BCE) loss. This guides the heads to learn distinct speaker activity patterns.
- Overlapped Speech Detection (OSD) Loss: Another head is constrained to match an OSD target mask using Mean Squared Error (MSE) loss, helping the model explicitly differentiate between non-speech, single-speaker, and overlapping speech regions.
The final objective function combines the standard diarization loss ($L_d$) with the weighted auxiliary losses ($L_S$ and $L_O$).
Datasets
The approach was evaluated on two primary benchmarks:
- Sim2spk: A simulated two-speaker dataset generated from Switchboard and NIST Speaker Recognition Evaluation corpora, featuring a high overlap ratio of 34.4% in training. Evaluation sets included varying overlap ratios (34.4%, 27.3%, and 19.6%).
- CALLHOME (CH): A real-world dataset consisting of two-speaker telephone conversations, used for model adaptation (155 mixtures) and evaluation (148 mixtures).
Key Insights and Ideas
- Targeted Regularization Beats Global Constraints: Instead of applying auxiliary losses to entire layers or adding complex residual connections (like previous architectures such as RX-EEND), targeting specific underperforming heads preserves the model's overall capacity while forcing diversification.
- Layer Depth Matters for Sub-tasks: Through empirical testing, the authors found that the SVAD loss yields the best results when applied to the final (4th) encoder block, whereas the OSD loss is most effective when applied to the first encoder block. This suggests that overlap detection is a low-level feature best captured early, while speaker-specific tracking requires high-level abstractions found deeper in the network.
- Visual Proof of Diversification: Visualizing the attention matrices confirmed that the proposed losses successfully broke up the identity-like diagonal lines, resulting in much richer, non-diagonal attention patterns that capture long-range speaker interactions.
Key Results
The proposed method achieved substantial performance gains over the baseline SA-EEND model across all test sets:
- Simulated Data (Sim2spk): At the default 34.4% overlap ratio, the Diarization Error Rate (DER) dropped from 6.30% (baseline) to 4.29% when combining both SVAD and OSD losses.
- Real Data (CALLHOME): The combined model achieved a DER of 8.67%, down from the baseline's 10.46%. This also outperformed the more complex RX-EEND state-of-the-art model, which scored 9.04% on their implementation.
Additionally, the ablation study proved that selecting heads based on their similarity to the identity matrix (using the trace heuristic) yielded superior results compared to randomly selecting heads for auxiliary losses.
Limitations and Future Work
While highly effective, the paper focuses exclusively on two-speaker diarization tasks. Extending this framework to an unknown or variable number of speakers remains an open challenge, as generating target masks for SVAD and OSD scales in complexity with more speakers. Furthermore, the hyperparameters balancing the loss functions ($\alpha$ and $\beta$) were fixed at 1; optimizing these dynamically could yield further performance improvements.
Summary updated