When two people talk at once, a speaker-diarization system (software that labels "who spoke when" in a recording) tends to fail. A standard system cuts the audio into short segments, turns each into a voice fingerprint called an embedding, and clusters the fingerprints so that each segment gets exactly one speaker. A segment with two voices produces a blended fingerprint that is moderately similar to both speakers. Clustering then either assigns it to one of them or invents a phantom third speaker. Network science has studied a problem with the same shape. It asks whether a person who is linked to two groups is a true bridge between separate communities, or just an ambiguous member of one loose community. The answer there is structural diversity: look at the person's neighbors and count how many mutually disconnected, internally cohesive groups they form. Signals that come from independent contexts count as independent evidence. Signals that come from one tight clique are redundant.
We propose to carry that test into diarization. For each audio segment we build the local graph of its nearest acoustic neighbors and count how many separate, triangle-cohesive groups those neighbors form. A segment whose neighbors split into two cohesive groups that do not touch each other is flagged as overlapped speech and given both labels. A segment whose neighbors form one connected mass is treated as a single speaker, however ambiguous its raw similarities are. The method needs no training. It reads only the geometry of the recording's own embeddings, so it cannot inherit the overlap rate of simulated training data. That inherited rate is the main reason end-to-end neural diarizers overshoot on real conversations.
Core Claim
Given a fixed speaker-embedding extractor, a segment's higher-order neighborhood structure separates true two-speaker overlap from single-speaker acoustic ambiguity better than first-order similarity does. Here "first-order" means cosine margins to cluster centroids or entropy of cluster posteriors. "Higher-order" means how many disconnected cohesive groups the segment's nearest neighbors form. A detector built on that count gives better-calibrated overlap estimates on real data than models trained on simulated mixtures. Plugged into a clustering pipeline, it closes much of the overlap gap between single-label clustering and multi-label systems.
Why This Is Not Already Done
The closest work is OCDGALP. It builds a segment affinity graph, refines embeddings with a graph attention network, and uses a modified label-propagation algorithm (LPANNI) that lets a node keep several dominant labels. That is overlapping community detection, and it reaches 11.07% DER with oracle voice activity detection on DIHARD III. However, its overlap decision comes from iterated label votes of immediate neighbors. It never asks whether those neighbors are connected to one another. The authors also report sensitivity to the fusion weight $\epsilon$ and the path length $\beta$.
Online graph-based label generation keeps a segment similarity graph and uses neighborhood likelihood to resolve ambiguous segments, but it does not model overlap at all. The Hitachi-JHU DIHARD III system handles overlap by voting across five complete systems with DOVER-Lap, not by any topological reasoning.
On the network-science side, the method comes from two papers. Dong et al. count connected components among the common neighbors of a node pair and gain 57% AUPR over plain common-neighbor counts for link prediction. Huang et al. define a "social context" as a maximal connected $k$-truss in an ego-network, which is a subgraph where every edge lies in at least $k-2$ triangles. They show this separates genuine contexts from loose sets of acquaintances better than components or cores do. Neither paper has been applied to acoustic affinity graphs. Dong et al.'s finding that the sign of the diversity effect differs between network families is also a constraint on us: we must show empirically that in speech graphs, diversity signals overlap rather than noise.
Method
For a recording, extract embeddings on short overlapping windows. We use a WavLM-based speaker encoder, because WavLM was pre-trained with simulated interfering speakers, which makes it the strongest available feature for this setting. Build a mutual $k$-nearest-neighbor graph $G$ on the speech segments. For segment $u$, let $N_k(u)$ be its neighbor set and $G[N_k(u)]$ the subgraph those neighbors induce with $u$ removed. The structural-diversity score is
$$ \mathrm{SD}_{\tau,m}(u) = \#\{\text{maximal connected } \tau\text{-trusses in } G[N_k(u)] \text{ with at least } m \text{ nodes}\}, $$
where $\tau$ sets how many triangles an edge needs to count as cohesive and $m$ discards tiny groups.
A single-speaker segment should score 1: its neighbors are same-speaker segments that are dense among themselves. A two-speaker segment should score 2: its neighbors come from pure-A and pure-B regions, and those regions are not linked to each other. Requiring triangles, rather than mere connectivity, is what stops a chain of ambiguous segments from joining two speakers into one context. This follows Huang et al.'s argument for trusses over components.
The pipeline then runs in three steps. First, cluster only the segments with $\mathrm{SD}=1$, using the same VBx or spectral backend as the baseline. Second, give each flagged segment the labels of the clusters that dominate each of its contexts. Third, as a secondary use of the same idea at the level of edges, down-weight edges whose endpoints' common neighbors fall into several components, following Dong et al. Such edges are bridges that would otherwise let clustering merge speakers.
Ego-networks hold only a few dozen nodes, and recordings hold thousands of segments. Truss decomposition at this scale is cheap, so the index structures of Huang et al. are not needed. We report runtime anyway, because OCDGALP left scalability to long, heavily overlapped recordings unexamined.
Research Program
The work has three parts. The first is a diagnostic study. On simulated mixtures with known overlap labels and on DIHARD III development data, we measure how well $\mathrm{SD}$ separates overlapped from clean segments. We compare it against first-order scores computed from the same graph and the same embeddings. We vary $k$, $\tau$, $m$ and embedding window length, and we report how flat the performance surface is, since hyperparameter sensitivity is OCDGALP's stated weakness.
The second part is the full diarization pipeline described above, evaluated end to end.
The third part is a calibration study. We run the detector on simulated data at several overlap ratios and on real corpora. We then check whether the fraction of segments it flags tracks the true overlap ratio of each corpus without retuning. The failure it targets is specific: EEND was trained on data with 27.3% overlap, over-predicts overlap on CALLHOME, which has 11.8%, and degrades from 12.28% DER in simulation to 31.01% on real data.
A small extension tests the score as a gate for target-speaker verification. EA-ASP masking lowers multi-speaker EER from 11.16% to 5.21%, but raises clean-speech EER from 1.13% to 1.60%. We apply the mask only when a test utterance's segments show $\mathrm{SD}\ge 2$ and ask whether this keeps both gains.
Baselines
The naive reference is the same embeddings, the same voice activity detection and the same VBx/spectral clustering with no overlap handling, so every segment gets one label. This is the classical modular pipeline described in NeMo's diarization system and in the DIHARD literature. Its missed-speech term in overlap regions is exactly what we aim to recover, and it has no reported result on Lacuna with our embeddings, so we will produce it ourselves.
The strongest single-system competitor is OCDGALP. It is the only published system that treats overlap as overlapping community detection on a segment graph, which is the question we are asking. It reports 11.07% DER with oracle VAD and 15.94% without on DIHARD III. We will quote those figures, and we will also reimplement GAT plus LPANNI on our embeddings and our graph so that the comparison isolates the decision rule rather than the embedding extractor. Both methods get the same DIHARD development-set search budget, measured in configurations tried. Beating it means lower DER than our reimplemented OCDGALP under oracle VAD and in the from-scratch condition, on the same embeddings, with the gain concentrated in overlap regions. Matching its DER while being training-free and less sensitive to hyperparameters would count as a partial win, and we would report it as such.
The strongest overall system in the file is the Hitachi-JHU DOVER-Lap ensemble: 11.58% DER on Track 1 Full EVAL and 16.94% on Track 2 Full EVAL. We treat it as a ceiling that uses roughly five times our compute rather than as a like-for-like competitor. We also test whether our system adds value as one more member of such an ensemble.
For end-to-end diarization we run EEND-EDA on CALLHOME. The file reports 23.07% DER for adapted EEND and 11.53% for x-vector clustering on CALLHOME, but gives no EEND-EDA figure, so we will measure it.
Two ablations isolate the key ingredient. The first replaces the truss-context count with a plain connected-component count, following Dong et al. The second replaces it with raw neighbor-label counts, which is essentially one step of label propagation. The first-order heuristics described below serve as a further ablation of the "higher-order" claim.
One caveat on comparability: the file does not say whether OCDGALP's DIHARD III numbers are on the Full or Core evaluation set. We will report both sets for every system we run ourselves.
Why not just…?
Why not just use end-to-end neural diarization, which handles overlap natively? EEND handles overlap by learning it from simulated mixtures, and that is where its weakness lies. On CALLHOME it reaches 31.01% DER unadapted and 23.07% adapted, against 11.53% for x-vector clustering (EEND). On DIHARD III it needed an ensemble to place second (Hitachi-JHU). Our calibration experiment directly compares EEND-EDA's predicted overlap fraction per corpus with ours. If EEND-EDA is well calibrated on real data and has lower DER, the argument from sim-to-real mismatch does not hold, and we will say so.
Why not just rely on WavLM features? We do use them. WavLM's pre-training with simulated interfering speakers improves the embeddings, but it still produces one vector per window and says nothing about how many speakers that vector represents (WavLM). The single-label baseline built on WavLM embeddings measures exactly how much overlap error remains once better features are in place.
Why not just threshold the similarity margin to the two nearest cluster centroids, or the entropy of cluster posteriors? This is the real threat. A blended embedding sits between two centroids, and a margin threshold may catch most overlap. Our hypothesis is that the margin cannot tell a true mixture from a single segment of one of two similar-sounding speakers, or from a noisy segment. Structural diversity can, because in those cases the neighbors are themselves ambiguous and link to each other. The margin also depends on a first-pass clustering that overlap has already corrupted. The diagnostic study settles the question. At matched recall, we compare overlap-detection precision of $\mathrm{SD}$, centroid margin, posterior entropy, and a logistic combination of all three, stratified by speaker-pair similarity. If $\mathrm{SD}$ adds nothing on top of the margin, the core claim fails.
Why not just use OCDGALP's label propagation, which already assigns multiple labels? Label propagation decides from neighbor label votes. Two neighbors voting "A" and "B" look the same whether they are distinct speakers or two noisy members of one merged cluster. The truss-versus-label-count ablation, run on the same graph, measures whether checking cohesion among neighbors matters. The hyperparameter sweep measures whether it also removes the sensitivity OCDGALP reports.
Decisive Checks
The data are DIHARD III (Tracks 1 and 2, Full and Core EVAL, with the web-video and restaurant domains reported separately, since their median DER is 35–45% even with oracle VAD), CALLHOME, VoxConverse, and simulated mixtures with controlled overlap ratios. For VoxConverse we found no reported DER on Lacuna for the graph-based systems in the file, so we will establish our own baselines there. We report DER split into missed speech, false alarm and confusion, together with JER, overlap-detection precision and recall, and predicted-versus-true overlap ratio per corpus.
Three results would decide the proposal. First, $\mathrm{SD}$ must add precision beyond centroid margin and entropy at matched recall on real DIHARD data; this is the necessary condition. Second, the full pipeline must beat our reimplemented OCDGALP on the same embeddings, with the gain coming from overlap-region missed speech rather than confusion elsewhere. Third, the flagged-overlap fraction must track true overlap from CALLHOME's 11.8% up to heavily overlapped simulation without retuning, where EEND-EDA does not. If the first check fails, we stop and report a negative result. That result would still show that first-order similarity already captures what neighborhood topology offers in speech graphs.
References
- Speaker Diarization with Overlapping Community Detection Using Graph Attention Networks and Label Propagation Algorithm
- Online Speaker Diarization with Graph-based Label Generation
- The Hitachi-JHU DIHARD III System: Competitive End-to-End Neural Diarization and X-Vector Clustering Systems Combined by DOVER-Lap
- Structural Diversity and Homophily: A Study Across More Than One Hundred Big Networks
- Truss-Based Structural Diversity Search in Large Graphs
- WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processing
- End-to-End Neural Speaker Diarization with Permutation-Free Objectives
- Enroll-Aware Attentive Statistics Pooling for Target Speaker Verification
- NeMo Open Source Speaker Diarization System