Graph-based speaker diarization works like this. Diarization is the task of deciding "who spoke when" in a recording. Each short speech segment becomes a node in a graph, each node is linked to its most similar segments, and a graph neural network learns which links join two segments from the same speaker. The weak point is the first step. Every segment gets the same fixed number of neighbors $k$, whether the room holds two people or twelve. A quiet participant with only a handful of segments is then forced to send most of their links into louder speakers' clusters. That is where their identity starts to leak away.
We propose to replace the fixed neighbor count with a local geometric test borrowed from dynamic-graph learning. We first build an over-connected graph. We then score each link by its Forman-Ricci curvature, a count-based measure of whether the two endpoints share many common neighbors (positive, "inside a community") or connect otherwise unrelated regions (negative, "a bridge"). Finally, we keep only the links that are not bridges. The effective neighborhood size then shrinks around minority speakers and grows inside large clusters on its own. The test is local, so it can also be updated cheaply as new audio arrives. The bet is that this one change removes much of the speaker-absorption failure that makes error rates climb with speaker count, and does so without per-dataset tuning.
Core Claim
The claim is narrow and testable. Prior work documents a large part of the density-scaling problem in clustering-based diarization: small speakers merged into large ones and speaker counts underestimated (Talker Diarization in the Wild). We hypothesize that much of this damage is done by the graph's construction, before any learning happens. It is not only a matter of embedding quality.
Signed curvature on an over-complete similarity graph should separate within-speaker links from cross-speaker links better than raw cosine rank. The reason is that a quiet speaker's spurious links into a large cluster have few shared neighbors (few triangles) and end at high-degree nodes, which is exactly the signature of negative Forman curvature. Pruning these links before message passing should stop the graph network from averaging the minority speaker's features into the majority, which is a known failure of shallow message passing. Gains should therefore grow with speaker count and be concentrated on low-talk-time speakers.
Prior work plays different roles here. The supervised graph clustering method SHARC and its end-to-end and overlap-aware extensions (SHARC; E-SHARC) supply the method we build on. They connect segments by $k$-nearest-neighbor cosine similarity, train a GraphSAGE network to predict node density and same-speaker link probability, and merge clusters hierarchically. Their authors note that $k$ and the merge threshold must be tuned per dataset. That is the gap we target, and we keep everything after graph construction unchanged.
TRicci (Edge Sparsification via Temporal Forman-Ricci Curvature) also supplies a method. It sparsifies dense temporal graphs by curvature combined with a recency decay and a penalty for crowded, redundant interactions, keeping the top 20% of edges while preserving 97.7% of link-prediction ROC-AUC. We reuse its scoring idea in a domain where the graph's communities are the prediction target. We also change two things. Retention is decided by curvature sign rather than a global fraction. The temporal term is learned rather than fixed, because in turn-taking speech, temporal closeness predicts same-speaker in calm dialogue and different-speaker in crosstalk.
The session-level graph convolutional network of Wang et al. motivates the approach. It refines embeddings by message passing over a globally thresholded similarity graph and reports 7.24% diarization error rate (DER) on CALLHOME, which shows that graph structure matters. However, it prunes with a single global scalar, which is the design we argue fails under density variation.
Research Program
Offline component. For each recording we build a graph connecting every segment to its $k_{\max}$ nearest neighbors, with $k_{\max}$ set generously (for example 60) and not tuned per dataset. For each link $(i,j)$ we compute a score
$$s_{ij} = F_{ij} + \alpha\, g_\theta(|t_i - t_j|, r_{ij}) - \beta\, c_{ij},$$
where:
- $F_{ij}$ is the triangle-augmented Forman curvature on the cosine-weighted graph, built from the endpoint degrees and the number of shared neighbors;
- $g_\theta$ is a small learned function of the time gap between the segments and the local turn-change rate $r_{ij}$;
- $c_{ij}$ is TRicci's competition term, which counts near-duplicate links from temporally adjacent segments;
- $\alpha$ and $\beta$ are weights fit once on training data.
We keep links with $s_{ij} \ge 0$, giving each node a data-dependent degree. The SHARC or E-SHARC GraphSAGE network is then trained and run on the pruned graph exactly as published. Curvature is not differentiable with respect to which links are kept, so the network is trained on graphs pruned by the fixed rule. Only $\theta$, $\alpha$ and $\beta$ are fit, by validation DER on a small grid.
Streaming component. A new segment adds links only to its own neighbors, so triangle counts, and hence curvature, change only for links within two hops of it. We update curvature in that region and re-run the graph network only there. Following Coden's lazy-update idea (Coden), we skip the re-run when the accumulated change in that neighborhood's curvature stays below a threshold. Speaker labels are kept stable across updates with Hungarian matching, as in the online systems of Zhang et al. (Online Speaker Diarization with Graph-based Label Generation; Low-Latency Online Speaker Diarization). Those systems report that their unpruned speaker-embedding graph becomes the slowest part on long recordings, and this component targets that bottleneck.
Out of scope. We deliberately set aside the deep, stable message passing of Anti-Symmetric DGN (A-DGN) and the spectral reshaping of AdaSpec (AdaSpec). SHARC's merging converges in at most three levels, so depth is not the bottleneck. AdaSpec is built for spectral filters and does not transfer to the spatial GraphSAGE network we use. We also do not claim to solve overlap among three or more speakers, which E-SHARC-Overlap leaves open. Overlap handling stays as in E-SHARC.
Baselines
Naive reference. The do-nothing point is the conventional x-vector + PLDA + agglomerative hierarchical clustering pipeline, run on the same embeddings and voice-activity segments. The SHARC paper reports a 53% relative improvement over this pipeline on AMI multi-distant-microphone recordings (SHARC). It anchors how much of any gain is due to graph learning at all.
Strongest existing method. This is E2E-SHARC, and E-SHARC with VBx resegmentation where overlap matters. It is the strongest competitor because it is the only graph diarizer in the file shown to scale to 21 speakers on VoxConverse, and our method differs from it only in graph construction. Reported numbers are as follows:
- AMI multi-distant microphone: 2.29% DER for E2E-SHARC (SHARC).
- AMI in the overlap-aware setting: 17.99% DER, falling to 17.21% with VBx (E-SHARC).
- VoxConverse: only a 44.4% relative improvement over its baseline is quoted, with no absolute DER found on Lacuna for SHARC.
- MSDWild and AliMeeting: no reported SHARC result found on Lacuna.
We will retrain SHARC with $k$ and the merge threshold tuned on each dataset's development set, using the same grid budget we spend on $\alpha$, $\beta$ and $\theta$. The same embeddings, segment lengths and GPU budget are used for both. Beating it means lower DER on the speaker-count strata above five speakers with no regression below that, using a single cross-dataset setting for our method against SHARC's per-dataset tuning.
Other references. On DIHARD III we also report MS-VBx, which reaches 18.2% DER (MS-VBx). On MSDWild's many-talker split we compare against the best published DER of 25.86% (MSDWild). For streaming, the reference is the low-latency graph-label system, which reports 6.60% DER on VoxConverse and 19.57% on DIHARD III (Low-Latency Online Speaker Diarization).
Ablations. The key ablation isolates curvature. We compare against three alternatives, each keeping exactly the same number of links per recording as our rule:
- random pruning;
- cosine-rank pruning, which keeps the strongest links;
- curvature alone, with $\alpha = \beta = 0$.
If cosine-rank pruning at matched sparsity does as well as curvature, the geometric argument fails.
Why not just…?
Why not just pick $k$ per recording, for example from an estimated speaker count or local density? This is the most serious threat, and the experiments are built to settle it. We run SHARC with three adaptive-$k$ variants:
- a per-recording oracle $k$ chosen with test labels, which serves as an upper bound;
- $k$ set from an eigengap speaker-count estimate;
- mutual-nearest-neighbor graphs with per-node density scaling.
A per-recording $k$ is still uniform within a recording. A meeting with one dominant speaker and three quiet ones needs different neighborhood sizes for different nodes, and only per-link pruning provides that. If oracle-$k$ SHARC matches our method on the minority-speaker metric, we will report that the idea reduces to choosing $k$ and drop the curvature claim.
Why not just let the graph network learn which links to ignore, since SHARC already predicts same-speaker probabilities? It does, but only after one or more rounds of message passing over the bad links. By then a minority speaker's features have already been averaged toward the majority. We test this with an iterative variant: run SHARC once, prune links with low predicted probability, and rerun. This shows whether learned self-pruning recovers the same gain at twice the cost.
Why not just use end-to-end neural diarization or VBx? End-to-end models have fixed output sizes that break beyond about four speakers (PRISM; MS-VBx). AliMeeting baseline DER rises from 4.84% with two speakers to over 23% with four (M2MeT). VBx was shown to add to graph clustering rather than replace it (17.99% to 17.21% on AMI in E-SHARC), so we apply VBx on top of both our system and SHARC rather than treat it as a rival.
Proposed Experiments
We evaluate on VoxConverse (1–21 speakers), MSDWild's few-talker and many-talker splits, AliMeeting, AMI and DIHARD III. Every result is stratified by speaker count.
Metrics. We report DER, Jaccard error rate (a per-speaker measure that weights quiet speakers equally), and absolute speaker-count error. We add a minority-speaker recall: the fraction of reference speakers with under 5% talk time that are mapped to their own output cluster.
Decisive check one. The gap between our method and tuned SHARC should widen monotonically with speaker count. It should be carried mostly by minority-speaker recall and count error, not by boundary errors. If gains are flat across speaker-count strata, the density mechanism is wrong.
Decisive check two. As a diagnostic, we measure the fraction of retained links that are cross-speaker, against oracle labels, at matched sparsity for curvature versus cosine rank. Curvature must be measurably purer, specifically around low-talk-time speakers.
Decisive check three. For streaming on long VoxConverse and DIHARD III files, we report real-time factor and DER against the low-latency online system. We check that incremental curvature updates keep per-segment cost flat as recordings grow, where the unpruned graph's cost rises.
References
- Talker Diarization in the Wild: the Case of Child-centered Daylong Audio-recordings
- Supervised Hierarchical Clustering Using Graph Neural Networks for Speaker Diarization
- Overlap-aware End-to-End Supervised Hierarchical Graph Clustering for Speaker Diarization
- Edge Sparsification via Temporal Forman-Ricci Curvature for Dynamic Graph Learning
- Speaker Diarization with Session-Level Speaker Embedding Refinement Using Graph Neural Networks
- Coden: Efficient Temporal Graph Neural Networks for Continuous Prediction
- Online Speaker Diarization with Graph-based Label Generation
- Low-Latency Online Speaker Diarization with Graph-Based Label Generation
- Anti-Symmetric DGN: a stable architecture for Deep Graph Networks
- AdaSpec: Adaptive Spectrum for Enhanced Node Distinguishability
- Multi-Stream Extension of Variational Bayesian HMM Clustering (MS-VBx) for Combined End-to-End and Vector Clustering-based Diarization
- MSDWild: Multi-modal Speaker Diarization Dataset in the Wild
- PRISM: Pre-trained Indeterminate Speaker Representation Model for Speaker Diarization and Speaker Verification
- M2Met: The Icassp 2022 Multi-Channel Multi-Party Meeting Transcription Challenge