Tuning EM by Basin Agreement: Choosing Hyperparameters Where Restarts Agree

Many clustering and sequence models are fit with expectation-maximization (EM), a two-step loop. One step guesses the hidden labels, such as which speaker is talking in each second of audio. The other step re-fits the model to those guesses. EM has two well-known weaknesses. It stops at whichever local solution it reaches first, and it carries knobs, such as how much to smooth or how strongly to trust the acoustic evidence, that are usually tuned once on an annotated development set and then frozen. These two weaknesses are normally treated separately. We propose to use one to fix the other. The knobs change the shape of the objective EM is climbing, so they decide how many competing solutions exist. For each new recording, we would choose the knob values at which EM started from several different sensible initial guesses ends up at the same answer. We would also require that the answer is not the trivial one of a single speaker.

Speaker diarization is the task of labelling "who spoke when" in a recording. It is a good testbed because both failures are documented there. Variational Bayesian HMM clustering is a probabilistic clustering method that adds a model of how speakers take turns. In its multi-stream form (MS-VBx), it beats agglomerative clustering, but it is initialized from that clustering and its scaling and regularization constants are hand-tuned on development data. Fixed-parameter systems also degrade badly on noisy domains of DIHARD III, with median error of 35–45% even in Track 1. If the agreement criterion works, it would give per-recording tuning of an iterative estimator with no development labels. It would also give a direct readout of whether a given fit sits in a well-defined basin.

Core Claim

Consider a single-speaker-stream VBx diarizer that clusters x-vectors, which are fixed-length speaker-identity embeddings computed from short audio segments. Its hyperparameters are written $\lambda$. They comprise the acoustic scaling factor, the speaker regularization constant, and the probability of staying with the same speaker from one step to the next. Each $\lambda$ induces a landscape over speaker assignments. When the model is over-flexible for a given recording, the landscape fragments into many basins, meaning regions of the objective that each pull EM toward a different fixed point. Different initializations then converge to different fixed points. When the model is over-smoothed, everything collapses into one or two speakers. Our claim is that the per-recording $\lambda$ maximizing a score that rewards both agreement and non-triviality tracks the $\lambda$ that a development-set oracle would pick for that recording.

The score is computed from $R$ structured initializations: NME spectral clustering at several neighbor counts, agglomerative clustering at several thresholds, and spherical k-means. Each is run to convergence under $\lambda$, giving final speaker labelings $z_1,\dots,z_R$. The score is

$$S(\lambda) = \Big[\tfrac{2}{R(R-1)}\sum_{i<j}\mathrm{AMI}(z_i, z_j)\Big]\cdot \bar H(\lambda),$$

where AMI is adjusted mutual information between two labelings, so 1 means identical and 0 means chance-level agreement. $\bar H(\lambda)$ is the average entropy of the speaker-time distribution across runs, which scores a one-speaker answer as zero. We select $\hat\lambda = \arg\max_\lambda S(\lambda)$ from a fixed grid and output the converged labeling with the highest evidence lower bound (the objective VB-EM maximizes) under $\hat\lambda$.

We pick hyperparameters this way rather than by maximizing that same evidence bound directly. The acoustic scaling factor tempers the likelihood, so evidence values computed under different scalings are not on a common scale. Agreement between fixed points is unit-free.

Why this is different from what exists

Auto-tuning spectral clustering with the normalized maximum eigengap tunes the affinity threshold $p$ and the speaker count per recording by minimizing $p/g_p$, the threshold divided by the largest normalized eigengap. It stops at a single partition and never runs an iterative estimator. Here we use its partitions only as some of the starting points, and we tune the iterative model's own continuous parameters.

Shum et al. run iterative Viterbi/GMM re-estimation from one spectral or k-means seed. They note that bad seeds get reinforced. We turn that disagreement between seeds into the tuning signal instead of treating it as a failure to hide.

The global-search work on the local-optimum problem goes the opposite way. Hybrid genetic search with shrinkage and stochastic component selection both try harder to find the best basin under fixed hyperparameters. We instead change the hyperparameters until the basins merge, which costs a few cheap restarts rather than an evolving population.

Theory both motivates and constrains the design. Jin et al. prove that mixtures with three or more components have arbitrarily bad local maxima. They also show that random initialization falls into them with probability at least $1-e^{-cM}$, where $M$ is the number of components and $c>0$ is a constant. This is why our starting points are structured rather than random, and why disagreement is informative. Dwivedi et al. show that under some forms of misspecification EM contracts to a unique projection from any start. This supports the reading of "all starts agree" as "the model is inside a contraction regime". Their result is univariate and two-component, so our link is an empirical hypothesis, not a theorem.

The scenario-dependent DIHARD-III system adapts per recording through a discrete domain classifier. That classifier fell from 100% to 80.7% accuracy on evaluation data. Our adaptation is continuous and uses no classifier.

Research Program

We first validate the mechanism on synthetic Gaussian mixtures generated with the ClusterGeneration setup of Sampaio et al.. We vary separation, dimension, and component count. The tuned knobs are the covariance shrinkage intensity $\alpha$ and a likelihood tempering factor. Because ground truth is known, we can measure directly whether $\arg\max S$ lands near the adjusted-Rand-optimal $\lambda$. We also check whether low $S$ flags the bad local maxima predicted by Jin et al.

We then move to diarization with x-vectors, oracle speech activity, and single-stream VBx. Oracle activity isolates clustering from detection errors, which DIHARD III attributes about 5% DER to. We compute per-recording oracle $\lambda$ curves, defined as the error obtained at every grid point using reference labels, and ask how well $S(\lambda)$ ranks them. If this holds, we would extend the criterion to the multi-stream MS-VBx setting, where the constants are currently hand-tuned. Throughout, we record compute in VBx iterations.

Baselines

The naive reference is VBx with default hyperparameters, initialized from agglomerative clustering. This is the standard pipeline, and every gain is measured against it.

The strongest practical competitor is VBx with $\lambda$ tuned by grid search on the DIHARD III development set and the CALLHOME adaptation split. It gets the same grid and the same total number of VBx runs, spent on development data instead of per recording. This comparison is decisive: if per-recording agreement cannot beat a globally tuned constant, the idea fails.

The strongest published method on this question is MS-VBx. The file reports 18.2% DER on DIHARD III against 19.3% for its constrained agglomerative baseline (MS-VBx). Its operating conditions differ from ours: it has an end-to-end frontend and handles overlap. So we compare against it only in the final multi-stream extension, using its frontend. On CALLHOME the reference is NME spectral clustering, which reports a 7.29% speaker error rate with oracle speech activity (NME-SC). We run it on our identical x-vectors.

Two further baselines isolate the ingredients. The first is multi-start VBx at the development-tuned $\lambda$ that keeps the highest-evidence run. It uses the same restarts but no hyperparameter selection, which isolates the value of choosing $\lambda$. The second ablation drops the entropy term from $S$, to show that it is what prevents collapse to one speaker.

Beating the development-tuned baseline means lower mean DER on DIHARD III evaluation, with the largest gains in the noisy domains. It also means that a paired per-recording test shows the gain is not confined to a few files.

Why not just…?

Why not just tune once on development data? This is the real threat. VBx with sensible global constants is strong, and if recordings rarely need different $\lambda$, our method only adds cost. The per-recording oracle analysis settles it. If the spread of oracle-optimal $\lambda$ across recordings is small, we will report that the idea is unnecessary. The DIHARD III domain variance and the 17% relative gain of NME-SC over development-tuned clustering suggest the spread is substantial, but neither result measures VBx specifically.

Why not just initialize VBx from NME-SC and stop? A better single starting point addresses local optima but leaves $\lambda$ fixed. The multi-start ablation and the NME-initialized single-run baseline measure exactly how much of any gain comes from initialization alone. If those two baselines match the full method, the hyperparameter selection is not earning its keep.

Why not pick $\lambda$ by the evidence lower bound, or the best-evidence restart? Tempering makes evidence values incomparable across acoustic scaling factors. We will still run evidence-based selection as a baseline, so the claim is tested empirically rather than asserted.

Isn't this just stability-based model selection? In spirit it is close. Stability methods choose cluster counts by robustness to resampling the data. Our version measures agreement among EM fixed points under structured initializations, and it tunes continuous parameters of a temporal model. We acknowledge that stability criteria are known to favour coarse solutions. The entropy ablation, and a sensitivity sweep over grid range, test whether we have avoided that failure.

Decisive Checks

Three checks decide the proposal. On synthetic mixtures, the rank correlation between $S(\lambda)$ and true adjusted Rand index should be clearly positive across separation levels. If it is not, the mechanism is wrong. On DIHARD III with oracle speech activity, agreement-tuned VBx should beat development-tuned VBx at an equal VBx-run budget, especially in the restaurant and web-video domains. The selected $\lambda$ should also correlate with the per-recording oracle $\lambda$. Finally, recordings with low maximum $S$ should have high error. This would make the score usable as an unsupervised failure flag even where it does not improve DER.