Speakers Unembedded: Embedding-free Approach to Long-form Neural Diarization

Xiang Li, Vivek Govindan, Rohit Paturi, Sundararajan Srinivasan

Introduction

Speaker diarization—the process of partitioning an audio stream into homogeneous segments according to speaker identity—is a fundamental task in speech processing. While modern End-to-End Neural Diarization (EEND) models have significantly improved performance by handling overlapping speech as a multi-label classification problem, they face inherent scalability issues. Standard EEND models are typically constrained to short audio segments and a fixed, small number of speakers.

To address long-form audio, traditional systems often resort to a "cascaded" architecture: detecting speech, extracting d-vector embeddings (mathematical voiceprints), and clustering these embeddings. However, this approach introduces error propagation and requires maintaining a separate, complex embedding module. Researchers at AWS AI Labs have introduced a "Local-global EEND" framework that eliminates the need for separate embedding models, repurposing the EEND architecture itself for both local segmentation and global speaker matching.

The Local-Global EEND Framework

The proposed system operates in three distinct phases, leveraging the EEND model's internal representations to link speakers across time without explicit biometric fingerprints.

1. Local EEND Processing

The input audio is divided into 30-second "local windows." A standard EEND model processes these windows to identify speaker segments locally. This step provides high-resolution diarization within short contexts but lacks the ability to track speakers across window boundaries.

2. Global EEND for Pairwise Matching

To establish global identities, the system identifies speaker chunks from the local phase and pairs them. These pairs are fed back into the same EEND model. By concatenating sequences from different windows, the model acts as a similarity scorer. It generates "inter-window posteriors"—scores indicating the likelihood that two chunks belong to the same speaker. This "EEND-as-a-scorer" approach is a key innovation, as it utilizes the model's learned feature space for matching rather than relying on external embedding extractors.

3. Affinity Matrix and Clustering

The similarity scores are used to construct an affinity matrix. A critical refinement in this stage is the "cannot-link" constraint: if two speaker chunks appear in the same 30-second local window, they are guaranteed to be different individuals. The system sets their similarity to zero in the matrix, effectively pruning potential errors. Spectral clustering is then applied to the matrix to resolve global speaker identities.

Overview of the Local-global EEND framework, illustrating local windowing (a), global pairwise matching (b), and affinity matrix construction with constraints (c).

Experimental Setup and Datasets

The model was trained on large-scale simulated mixtures derived from standard corpora, including Switchboard-2, Switchboard Cellular, and NIST Speaker Recognition Evaluation (2004-2008). To improve the global matching phase, the researchers fine-tuned the model on concatenated sequences ($EEND_{concat}$), teaching the network to distinguish between different speakers even when their speech is artificially joined.

Evaluation was conducted on several benchmarks, including CALLHOME (2-6 speakers), Callhome American English (CHAE, ~30 min recordings), and RT03-CTS (~10 min recordings).

Results and Performance

The Local-global EEND approach demonstrated substantial improvements over baseline 1-pass EEND models. On the CHAE dataset, the framework achieved a 13% relative reduction in Diarization Error Rate (DER). On RT03-CTS, it achieved a 10% reduction.

One of the most significant findings was the system's ability to scale to high speaker counts. As shown in the table below, the $EEND_{concat}$ model significantly outperformed the vanilla EEND in sessions with 5 or 6 speakers, where traditional EEND models often struggle or fail entirely.

Comparison of Diarization Error Rate (DER) for 1-pass and Local-global EEND models across varying speaker counts.

Beyond accuracy, the researchers optimized the system for real-world deployment. By implementing batching for the global matching step and sampling a subset of frames (as few as 64) from each speaker chunk, they reduced the Real Time Factor (RTF) by nearly 70% with negligible impact on DER.

Discussion and Conclusion

The Local-global EEND framework represents a shift toward simpler, more integrated diarization systems. By removing the separate embedding module, the architecture reduces complexity and mitigates the privacy risks associated with storing persistent voice biometrics. Instead, the system relies on relative similarity within a single session.

While the system shows a slight performance dip in 4-speaker sessions—likely due to window size sensitivity—its overall robustness in long-form, multi-speaker scenarios is a significant advancement. Future work will likely focus on dynamic windowing and training on even larger datasets to further refine the model's discriminative power.

Summary updated

BibTeX
@inproceedings{li2024speakers,
  author = {Xiang Li and Vivek Govindan and Rohit Paturi and Sundararajan Srinivasan},
  title = {{Speakers Unembedded: Embedding-free Approach to Long-form Neural Diarization}},
  booktitle = {Interspeech},
  year = {2024},
  doi = {10.21437/interspeech.2024-1174},
}
Figure