Introduction
Speaker Diarization (SD) is the task of partitioning an audio stream into segments based on speaker identity, effectively answering the question "who spoke when?" In modern speech processing pipelines, this is typically handled via a modular approach: first, an embedding extractor (such as an x-vector network) transforms audio segments into high-dimensional vectors; second, an unsupervised clustering algorithm (such as Agglomerative Hierarchical Clustering (AHC) or Spectral Clustering) groups these vectors into speaker-specific clusters.
However, the reliance on unsupervised clustering presents a significant bottleneck. These algorithms operate on fixed mathematical heuristics—such as merging the two closest points in a vector space—without considering the underlying distribution of the specific dataset. While End-to-End Neural Diarization (EEND) models have emerged to address this, they often struggle with scalability, particularly in scenarios involving a high number of speakers. To bridge this gap, researchers from IISc Bangalore have introduced SHARC (Supervised HierArchical gRaph Clustering), a novel framework that replaces traditional clustering with a supervised Graph Neural Network (GNN).
The SHARC Framework
SHARC re-imagines the clustering problem as a graph-based supervised learning task. The process begins by representing audio segment embeddings as nodes in a graph, with edges initially established between each node and its $k$-nearest neighbors based on cosine similarity.
GNN-Based Prediction
At the core of SHARC is a GraphSAGE module, a GNN architecture designed to learn inductive representations. Unlike traditional clustering, the GNN is trained to predict two critical components:
- Edge Linkage Probability ($p_{ij}$): The probability that two connected nodes belong to the same speaker.
- Node Density ($d_i$): A metric indicating how representative or "central" a node is within its local neighborhood.
Hierarchical Merging Logic
During inference, SHARC utilizes these predictions to perform iterative merging. Instead of simple distance metrics, the algorithm identifies high-density nodes as potential cluster centers and merges neighboring nodes into them if the predicted linkage probability exceeds a specific threshold ($p_\tau$). This hierarchical process repeats until the graph converges, typically within a few iterations ($M \le 3$). By using supervised density and linkage metrics, the model effectively ignores outliers and noisy segments that often trip up unsupervised methods.
E2E-SHARC: Joint Optimization
The authors further extend this concept with E2E-SHARC, an end-to-end variant that jointly trains the embedding extractor (an ETDNN model) and the GNN clustering module. This joint optimization ensures that the embedding extractor learns to produce feature vectors specifically tailored for the GNN’s clustering logic. This synergy allows the system to minimize the Diarization Error Rate (DER) more effectively than when treating the components as independent black boxes.
Experimental Results
The SHARC framework was evaluated on two prominent benchmarks: the AMI meeting corpus (3-5 speakers) and the Voxconverse dataset (1-21 speakers).
Performance Benchmarks
- AMI Dataset: E2E-SHARC demonstrated a remarkable 53% relative improvement over traditional x-vector + PLDA + AHC baselines. On the AMI Multi-Distant Microphone (MDM) task, it achieved a state-of-the-art DER of 2.29%.
- Voxconverse Dataset: The system achieved a 44.4% relative improvement over the baseline. Notably, SHARC maintained robust performance even as the number of speakers increased to 21, a scenario where many end-to-end models typically fail.
- Refinement: The study also showed that SHARC provides a superior initialization for temporal refinement techniques like VBx, leading to even lower error rates when used in tandem.
Discussion and Limitations
While SHARC represents a significant leap in supervised diarization, it introduces new considerations. The model's performance is sensitive to hyperparameters such as the number of neighbors ($k$) and the linkage threshold ($p_\tau$), which may require tuning for different acoustic environments. Additionally, while the hierarchical graph approach is efficient, it is computationally more complex than simple K-means clustering.
Future research directions include the integration of explicit overlap detection to better handle segments where multiple speakers talk simultaneously, and further optimizing the GNN architecture for real-time processing applications.
Conclusion
SHARC demonstrates that introducing supervision into the clustering stage of speaker diarization can yield substantial performance gains. By leveraging the relational power of Graph Neural Networks and the stability of density-based merging, SHARC sets a new standard for modular diarization systems, particularly in complex, multi-speaker environments.
Summary updated