When two people talk at once, a speaker diarization system (software that answers "who spoke when" in a recording) has to decide which two of its known speakers produced the overlapped stretch. Current systems handle this poorly. Most either drop the overlap or give it to whichever speaker cluster sits nearest. We argue that how hard this decision is depends mainly on one measurable quantity: how far apart the two speakers' habitual pitch registers are. The most common training corpora are heavily male. So when the two voices sit close in pitch, and especially when both are female, systems may be making their least reliable decisions exactly where they have seen the least data. We propose to measure this directly and then fix it with a small, closed-form statistical model. The model scores each candidate speaker pair for an overlapped segment and becomes appropriately uncertain when the pair's pitch gap is small.
The opportunity is that three facts in the literature have never been put together. First, the standard way to score speaker similarity, probabilistic linear discriminant analysis (PLDA, a Gaussian model that splits a speaker embedding into an identity part and a nuisance part), assumes each embedding comes from exactly one speaker. Second, pitch is the main physical cue that separates concurrent voices. Third, dataset gender skew is well documented for single-speaker recognition but has never been audited inside overlapped conversation. Our contribution is a two-identity PLDA whose residual noise depends on the pitch gap. It is paired with the first audit of overlap-attribution error stratified by pitch gap and by the gender makeup of the overlapping pair.
Core Claim
Our first claim is that speaker-attribution error inside overlapped regions grows as the pitch-register gap between the overlapping speakers shrinks. Our second claim is that, at a fixed gap, error is higher for female–female pairs than for male–male pairs when the embedding extractor and backend are trained on male-skewed data. Our third claim is that a PLDA backend which explicitly models an overlapped embedding as generated by two identities, with noise that widens as the pitch gap narrows, reduces this error and narrows the gender gap. We expect it to do better than simple nearest-cluster assignment or gender-balanced data alone. The pitch information enters at the level of each speaker cluster, estimated from that speaker's clean non-overlapped speech. We never track pitch inside the overlap, where trackers are known to fail through pitch halving and doubling (Pitch Contour Separation from Overlapping Speech).
Where this sits in prior work
Two papers motivate the gender-aware angle, and they pull in opposite directions. Mixture of PLDA models in i-vector space for gender-independent speaker recognition treats gender as a hidden variable and marginalizes it out. On NIST 2010 telephone males this improved old MinDCF from 0.112 to 0.096, but it operates only on single-speaker trials. Multi-PLDA Diarization on Children’s Speech fuses an identity PLDA with a coarse child/female/male PLDA. It found that balanced training of the coarse PLDA beats unbalanced training, but it used static category labels, oracle speech detection, and no overlap model. Our design sides with neither "hide gender" nor "label gender." It conditions on a continuous acoustic quantity, the pitch gap, and then tests whether gender still explains residual error once that quantity is controlled.
The Savoldi et al. study, Twists, Humps, and Pebbles: Multilingual Speech Recognition Models Exhibit Gender Performance Gaps, probes multilingual speech recognizers and reports that raw pitch and intensity do not explain gender error gaps. This apparently contradicts our premise. We address it head on. Their finding concerns transcribing a single speaker, where pitch is incidental to the task. In overlap attribution, pitch difference is part of the physics of the problem. Our audit is designed so that it could reproduce their "red herring" result in our setting: if pitch gap turned out not to predict attribution error, our method would lose its motivation.
Two papers supply the prosodic cues. The detection of overlapping speech with prosodic features for speaker diarization uses long-window pitch, intensity and formant statistics to detect overlap on the AMI meetings corpus, gaining up to 7.2% relative in diarization error rate. It uses prosody only to decide whether overlap occurs, not who is overlapping. Ishimoto et al. separate per-speaker pitch contours from simulated two-speaker mixtures and observe that their learned speaker embeddings cluster by gender. That is direct evidence that separation exploits pitch-register differences, but their method needs known speakers and was never connected to diarization.
On the overlap side, End-to-End Neural Speaker Diarization with Permutation-Free Objectives replaces clustering with a neural network that outputs several active-speaker flags per frame. Its results constrain our design. On real CALLHOME telephone calls it reached 23.07% error after adaptation, versus 11.53% for the x-vector clustering pipeline, which shows that the modular PLDA route remains competitive on real audio. DOVER-Lap: A Method for Combining Overlap-Aware Diarization Outputs combines system outputs by voting, reaching 20.5% on AMI. The DIHARD challenges (The Second DIHARD Diarization Challenge: Dataset, Task, and Baselines, The Third DIHARD Diarization Challenge) supply the x-vector/PLDA baselines, in which overlap is scored as missed speech. None of these reports errors broken down by demographic group or pitch gap.
The Model
Standard PLDA writes a single-speaker embedding as $w = m + \Phi y + \Gamma z + \epsilon$. Here $y$ is the speaker's latent identity, $\Phi$ spans the directions along which speakers differ, $\Gamma z$ captures session and channel variation, and $\epsilon$ is residual noise. For an embedding $w_o$ extracted from an overlapped segment, we add a second PLDA that shares the identity variables:
$$w_o = m_o + \Phi_o (y_a + y_b) + \Gamma_o z + \epsilon_o, \qquad \epsilon_o \sim \mathcal{N}(0, \Sigma(\delta)).$$
In this equation, $y_a$ and $y_b$ are the same identity variables used by the single-speaker PLDA. $m_o$, $\Phi_o$ and $\Gamma_o$ are separate parameters fitted to overlap embeddings, because the embedding of a mixture is not the average of the two speakers' embeddings. $\delta$ is the absolute difference, in semitones, between the two speakers' median pitch measured on their non-overlapped speech. Using the sum $y_a + y_b$ makes the model indifferent to the order of the pair. We parameterize $\Sigma(\delta)$ as a small interpolation between a few learned covariance matrices indexed by bins of $\delta$.
Every term is Gaussian, so scoring stays in closed form. Clustering first gives each speaker cluster $i$ a posterior over its identity $y_i$. For an overlapped segment, we then compute the likelihood of $w_o$ under each candidate pair $(i, j)$ and normalize across pairs to get a calibrated posterior over which pair was talking. When $\delta$ is small, $\Sigma(\delta)$ widens and the pair posteriors flatten. That flattening lets a downstream rule decline to assign a second label rather than guess. We train the model with EM (expectation–maximization) on simulated two-speaker mixtures in which both identities are known. The mixtures are built from the same data used to train the backend, and we deliberately control their gender composition and pitch-gap distribution.
Research Program
The work has three stages.
The first stage is an audit. We run existing overlap-aware pipelines on the AMI meetings corpus, DIHARD III, and CALLHOME, using an embedding extractor trained on VoxCeleb 1 & 2. For every reference overlap region, we compute the two speakers' pitch gap $\delta$ from their reference single-speaker regions. Where speaker gender metadata exists, we also label the pair's composition (female–female, male–male, mixed). Following the regression approach of FaiST: A Benchmark Dataset for Fairness in Speech Technology, we then fit a mixed-effects model of overlap-attribution error. Its predictors are $\delta$, pair composition, overlap duration and domain, with a random effect for recording. This tells us whether pair composition still predicts error once $\delta$ is controlled.
The second stage builds the model. We fit the two-identity PLDA on simulated mixtures. We then plug its pair posteriors into the second-speaker assignment step of the DIHARD-style pipeline, which runs x-vector extraction, PLDA scoring, agglomerative clustering, and resegmentation with a Bayesian hidden Markov model (VB-HMM).
The third stage tests how gender skew interacts with the model. We vary the gender balance of the simulated training mixtures (natural VoxCeleb-like skew versus balanced) and vary whether $\Sigma$ depends on $\delta$ alone or also on the pair's absolute pitch register. The second variant tests whether low-gap female pairs need their own statistics because they occupy a sparsely trained region of embedding space.
Baselines
The naive reference point is the DIHARD baseline pipeline as published: x-vectors, PLDA, agglomerative clustering, and VB-HMM resegmentation, with overlap left unassigned and therefore counted as missed speech. The Second DIHARD Diarization Challenge: Dataset, Task, and Baselines reports 25.99% evaluation error for this system on Track 1, where reference speech activity is given. We rerun it with our own extractor so that all systems share the same embeddings.
The heuristic overlap baseline keeps the same pipeline and the same overlap detector, but assigns each overlapped segment's second label to the nearest cluster other than the first, in time or by embedding. This is the simplest way to give overlap a second speaker, and it is exactly the decision our model replaces. Beating it is therefore the main test of our method.
The strongest modular competitor adds the prosodic overlap handling of Kockmann et al., which labels overlap with a second speaker and excludes overlapped frames when building speaker models. That paper reports up to 7.2% relative improvement in diarization error on AMI over a spectral-only overlap detector.
The strongest end-to-end competitors are EEND and DOVER-Lap. We train EEND on the same simulated mixtures and adapt it on the same development data; Fujita et al. report 23.07% on CALLHOME after adaptation. We apply DOVER-Lap on top of the systems above, using its released code; on AMI it is reported at 20.5%. For DIHARD III, the only published reference numbers are a median Track 1 error below 20% and winners roughly 7 points better than the baseline. We found no reported result on Lacuna for any system's overlap-attribution error broken down by pitch gap or gender, so we produce that number ourselves for every baseline.
All systems share the same training audio, the same simulated mixtures, the same overlap detector (and, in a separate condition, oracle overlap), and the same development-set tuning budget for thresholds and fusion weights.
The ablations isolate the key ingredient. First, the two-identity PLDA with a constant $\Sigma$ removes pitch-gap conditioning. Second, the full model is trained on skewed versus balanced mixtures. Third, the pitch-gap-conditioned model is compared with a variant that conditions on hard gender labels in place of $\delta$.
We count the method as beating the baselines if it meets three conditions. It must reduce overlap-attribution error in the lowest $\delta$ bin relative to nearest-cluster assignment and to Kockmann-style labeling, across recordings and not only in aggregate. It must shrink the female–female versus male–male gap at matched $\delta$. And it must not raise overall diarization error or Jaccard error rate on AMI, DIHARD III or CALLHOME.
Why not just…?
Why not just rebalance the training data by gender? This is a real threat. Balanced coarse PLDA training helped in Multi-PLDA Diarization on Children’s Speech. Rebalancing could close the female–female gap on its own, but it would not address the dependence of error on pitch gap, which also affects male–male pairs. The balanced-mixture ablation settles this. If balanced data with a constant $\Sigma$ matches the full model in the low-gap bin, we will report that the pitch-gap conditioning is unnecessary and reframe the contribution around the audit and the two-identity scoring.
Why not just give the overlap to the nearest other cluster? Nearest-cluster assignment is cheap and often right when one other speaker is clearly active nearby. It has no notion of when it is likely to be wrong, however, so it cannot abstain in low-gap overlaps. This heuristic is our primary baseline. If it matches our method in the low-gap bin, the generative model does not earn its place.
Why not just use end-to-end neural diarization, which handles overlap natively? EEND's overlap handling is learned from simulated mixtures, and on real telephone speech it trailed the clustering pipeline (23.07% versus 11.53% error on CALLHOME per Fujita et al.). It also gives no calibrated pair posterior. Its errors may still be lower in some domains, so we train it on the same gender-controlled mixtures and report it inside the same stratified audit. That comparison shows whether its errors carry the same pitch-gap and gender pattern.
Why not just separate pitch tracks first, as Ishimoto et al. do? Their separator needs a known target speaker and was validated only on simulated Japanese speech. Our model uses pitch only from clean single-speaker regions, which diarization already provides, so it does not depend on in-overlap pitch estimates that are unreliable on real audio.
Proposed Experiments and Decisive Checks
The first decisive check is the audit. In the mixed-effects regression on AMI and DIHARD III, the coefficient on $\delta$ must be negative and significant. If it is not, pitch gap is a red herring for overlap attribution, just as Savoldi et al. found it to be for recognition errors, and the modeling stage is dropped.
The second check is a controlled simulation. On mixtures built from held-out VoxCeleb speakers with matched $\delta$ distributions, we measure attribution error per composition. This separates the effect of the pitch gap from the effect of gender while the recording domain stays fixed.
The third check is the main comparison on real conversation. We report overall diarization error and Jaccard error rate with a 0 ms collar, overlap-attribution error per $\delta$ bin, and the female–female minus male–male gap at matched $\delta$ for every baseline and ablation. We run this with both detected and oracle overlap, so that errors in overlap detection are not mistaken for errors in speaker assignment.
The fourth check is calibration. Using the log-likelihood-ratio cost, we test whether the pair posteriors in the lowest $\delta$ bin are better calibrated than those from the constant-$\Sigma$ ablation. We also test whether allowing the system to abstain on flat posteriors reduces confusion errors more than it adds missed speech.
The method is supported only if the low-gap improvement and the gender-gap reduction both survive the balanced-data ablation.
References
- Pitch Contour Separation from Overlapping Speech
- Mixture of PLDA models in i-vector space for gender-independent speaker recognition
- Multi-PLDA Diarization on Children’s Speech
- Twists, Humps, and Pebbles: Multilingual Speech Recognition Models Exhibit Gender Performance Gaps
- The detection of overlapping speech with prosodic features for speaker diarization
- End-to-End Neural Speaker Diarization with Permutation-Free Objectives
- DOVER-Lap: A Method for Combining Overlap-aware Diarization Outputs
- The Second DIHARD Diarization Challenge: Dataset, Task, and Baselines
- The Third DIHARD Diarization Challenge
- FaiST: A Benchmark Dataset for Fairness in Speech Technology