Relational Timing for Speaker- and Rate-Invariant Speech Segmentation

Speech has no spaces between words, so a listener or a recognizer has to decide where one sound or word ends and the next begins. Two kinds of evidence help. The first is how the spectrum changes, which marks where sounds shift. The second is how long things last: a geminate stop, a doubled consonant held longer than its single counterpart, differs from a singleton mainly in duration, and "cap" is slightly longer than the "cap" inside "captain". Both kinds of evidence are distorted in predictable ways. A different voice warps the spectrum. A faster talker shrinks every duration. Current segmenters use raw spectral change and raw durations, so they are fragile when either the talker or the speaking rate differs from what they were tuned on.

We propose to build a segmenter in which both kinds of evidence are relational. Spectral boundary evidence comes from divergences between local sound distributions, which do not change when a speaker's cepstra undergo an affine warp. Duration evidence comes from the ratio of a candidate segment's length to a nearby anchor window of about two syllables, which does not change under uniform rate scaling. The claim is specific and testable. When test speech differs from training speech in speaking rate or speaker, a segmenter built on these two relational cues should keep its boundary accuracy, and a segmenter using absolute duration penalties and Euclidean spectral change should lose it. We also claim the local anchor matters: normalizing by global utterance rate should not be enough.

Core Claim

Two findings, from literatures that do not cite each other, point to the same design. The structural-representation work of Minematsu and colleagues showed that the Bhattacharyya distance between Gaussian sound distributions stays the same when every distribution passes through the same affine map $x \mapsto Ax+b$. A structure model trained on one male speaker therefore recognized isolated vowels from eight new speakers perfectly, as reported in Japanese vowel recognition based on structural representation of speech. On the timing side, the ratio of closure duration to word duration classified Japanese singleton and geminate stops at 97.5% across slow, normal and fast speech, even though absolute closures overlapped heavily (Durational structure of Japanese single/geminate stops). Perception experiments found that listeners anchor on the surrounding disyllable ($R^2=.990$ across rates; The perception boundary between single and geminate stops).

Both findings are statements about invariance to a nuisance transform, and neither has been used to find boundaries. We write the segment-level score of a candidate segmentation as

$$ S(\text{seg}) = \sum_{k} \Big[\, w_s \, D_B\big(\mathcal{N}_{k}^{-}, \mathcal{N}_{k}^{+}\big) \;+\; w_d \, \log p\big(\log d_k - \log \bar{d}_{A(k)}\big) \Big], $$

The symbols are as follows. $k$ indexes hypothesized boundaries. $\mathcal{N}_k^{-}$ and $\mathcal{N}_k^{+}$ are Gaussians fit to cepstral frames just before and after boundary $k$. $D_B$ is the Bhattacharyya distance. $d_k$ is the duration of the segment ending at $k$. $\bar d_{A(k)}$ is the mean segment duration inside a local anchor window $A(k)$ spanning roughly two syllables. $p$ is a density over log duration ratios learned without labels from the segmenter's own output. $w_s$ and $w_d$ are weights set on development data.

Each term is invariant for a simple reason. A shared affine warp leaves $D_B$ unchanged. Scaling all durations by a rate factor $\alpha$ adds $\log\alpha$ to both $\log d_k$ and $\log \bar d_{A(k)}$, so their difference is unchanged. The anchor is local because rate drifts within an utterance, and because the perception data favor a disyllable-sized anchor over the whole word.

How This Differs from Existing Work

The closest work is Discriminative reranking for LVCSR leveraging invariant structure. It scores f-divergence structures between phones for N-best rescoring and reports a 14.1% relative WER gain on Japanese digits. However, it takes phone boundaries from a baseline HMM, and it uses duration only as a length normalizer. Our proposal reverses both choices: the structure proposes the boundaries, and duration enters as a relational term. Automatic recognition of connected vowels extended structures to connected speech, but it sliced words into a fixed number of distributions. Without acoustic-substance time warping its accuracy fell from 92.6% to 66.6% (73.4% with non-substance warping). That result both motivates and constrains our design. It shows that alignment is the weak point of structural dynamics, and it warns that purely relational features can lose information. For that reason we keep the divergence strictly local, between adjacent windows, instead of matching whole-utterance matrices.

A Joint Spectro-Temporal Relational Thinking Based Acoustic Modeling Framework builds relational graphs over learned neural features for phone recognition and reaches 9.20% phone error rate on TIMIT. It does not segment speech, and it has no invariance guarantee. We borrow its observation that relational modeling helps most on long vowels, and we test whether short segments benefit less.

On the timing side, Using Duration Information in Cantonese Connected-Digit Recognition models absolute state durations with Gamma distributions. This cut insertions from 418 to 127 but increased deletions in fast speech, which is exactly the failure a rate-invariant term should remove. Using durational cues in a computational model of spoken-word recognition hand-sets durations in its lexicon, and the model has no speaker normalization. The unsupervised segmenters supply our search machinery: Towards Unsupervised Phone and Word Segmentation Using Self-Supervised Vector-Quantized Neural Networks and A Simple HMM with Self-Supervised Representations for Phone Segmentation both use dynamic programming with a scalar, absolute duration penalty. We reuse that dynamic-programming search unchanged and replace only the scoring terms.

Research Program

The system has three parts, and each can be swapped out independently. The first is a boundary-evidence function. We fit diagonal Gaussians to short windows of mel-cepstra on either side of each frame and compute the Bhattacharyya distance between them. A low-pass variant follows the 2 kHz filtering that helped in the vowel work. This replaces the Euclidean spectral variation function, meaning the frame-to-frame spectral change curve whose peaks marked boundaries in the HMM segmentation paper. The second part is the relational duration model. It is initialized from peak-picked boundaries, and its density $p$ is re-estimated over a few rounds of hard expectation-maximization, which alternates between segmenting the corpus and refitting the density to the resulting segments. The third part is dynamic-programming search over the combined score.

We will also test whether the same relational duration term, attached to a HuBERT-based segmenter, transfers across feature types. This separates the value of the duration term from the value of the divergence-based spectral term.

One analysis addresses the known over-invariance problem. Affine invariance can make distinct sounds look identical, and anchor normalization can hide a real lengthening, since a geminate also lengthens its word. We will measure how often relational scoring merges boundaries that absolute scoring keeps, and break the result down by phone class. We expect vowel-vowel and semivowel transitions to be hardest; this is where SCPC also reports failures.

Baselines

The naive reference point is peak picking on the Euclidean spectral variation function over mel spectrograms, with a fixed threshold and no duration model. The HMM segmentation paper reports that this simple method reaches up to 88.4% R-value on TIMIT, where R-value is a boundary score that penalizes over-segmentation (the r-value paper). Because 88.4% is the best TIMIT number in the file, this "naive" method is also the strongest competitor on matched conditions, and we treat it as such.

The strongest learned competitors are HMM-DP-BF (84.4% R-value on TIMIT; A Simple HMM with Self-Supervised Representations) and SCPC (87.44 R-value on TIMIT). On Buckeye, VQ-VAE with penalized dynamic programming reports phone-segmentation F-score 77.5% at over-segmentation 20.9 (Towards Unsupervised Phone and Word Segmentation). For recognition-style disambiguation on CUDIGIT, the reference is the absolute-state Gamma duration model at 97.45% digit accuracy with 127 insertions, compared with 95.09% and 418 insertions for the plain HMM.

All systems will be rerun by us on identical training and test splits. Each will get the same hyperparameter search budget (the same number of development evaluations for thresholds, duration weights and $\lambda$) and the same 20 ms boundary tolerance. We report the published numbers only as sanity checks on our reimplementations.

Two ablations isolate the key ingredient. The first replaces the local-anchor ratio with duration normalized by global utterance rate. The second replaces $D_B$ with Euclidean spectral change after cepstral mean normalization.

The criterion for beating the baselines is set in advance. On matched TIMIT we expect only to stay within about one R-value point of spectral-variation peak picking. The real test is the mismatched conditions described below. We count the proposal as successful if its R-value drops by at most half as much as the best baseline's, under both rate mismatch and speaker mismatch. On CUDIGIT, success means fewer deletions than the Gamma model on the fastest third of utterances without giving back its insertion gains. No reported result was found on Lacuna for any of these baselines under rate-shifted or cross-gender test conditions, so those numbers will be produced here.

Why not just…?

Why not just normalize durations by global speaking rate? This is the most serious threat, because a global rate ratio is itself a relational cue and costs almost nothing. The geminate evidence argues for a local anchor: the disyllable predicted listeners' boundaries at $R^2=.990$ within a word. Spontaneous speech also changes rate within a single utterance; embedded words are about 9% shorter in Dutch interviews but only 2.5% shorter in read speech (Durational information in word-initial lexical embeddings). Still, this evidence comes from perception and corpus statistics, not from segmenters. The global-rate ablation settles the question directly. If it matches the local anchor on Buckeye and on natural fast-versus-slow CUDIGIT subsets, we will report that the gain comes from relational duration as such, not from the anchor. The contribution then shrinks to a cheaper rate-normalized penalty.

Why not just apply cepstral mean normalization and keep Euclidean spectral change? Mean normalization removes the offset $b$ but not the scaling and rotation $A$ that vocal-tract differences introduce. The vowel study found that normalized HMMs still degraded under speaker mismatch, where structure models did not. The tone study found that normalizing the wrong statistics removed useful contrast: full syllable-level normalization hurt, and only a hybrid scheme reached 65.38% (Mandarin tone recognition using affine-invariant prosodic features). The second ablation measures this directly on cross-gender segmentation.

Why not just use HuBERT or wav2vec 2.0 features trained on many speakers? Large self-supervised models do average out speaker variation. However, the HMM segmentation paper shows that their contextual smoothing blurs boundaries, so mel-spectrum peak picking beats them. They also still need an explicit duration penalty, which is absolute. We therefore attach our relational duration term to a HuBERT segmenter. If HuBERT plus relational duration matches the full system, the divergence-based spectral term is unnecessary, and we will say so.

Why not just tune the absolute duration penalty $\lambda$ per condition? Retuning requires labeled data from the target rate, which a deployed segmenter does not have. The Cantonese results show that a single absolute model trades insertions for deletions as rate changes. The rate-shift experiment reports an oracle-tuned $\lambda$ per rate as an upper bound. If the relational term with one global setting closes most of the gap to that oracle, the claim holds.

Proposed Experiments

The decisive experiment is rate mismatch. We tune every system on normal-rate TIMIT and test on TIMIT audio time-scaled by 0.7× and 1.4×. We then repeat the test on natural rate variation by binning Buckeye and CUDIGIT utterances into speaking-rate terciles, since synthetic stretching does not reproduce the uneven compression of real fast speech.

The second experiment is speaker mismatch. We train and tune on male TIMIT speakers and test on female speakers, with and without vocal-tract-length perturbation applied to the test set.

The third experiment targets duration-dependent contrasts. We use CUDIGIT digit insertions and deletions, especially the digits "2" and "5", which account for nearly 70% of timing errors. We also use single-versus-geminate classification on Japanese multi-rate recordings in the style of Hirata and Amano. If the original 432 tokens cannot be obtained, we will re-record a matched set. There the relational model should approach the 97.5% ratio-based accuracy without being given hand-measured closure durations.

Across all three experiments we report R-value, F-score and over-segmentation, broken down by phone class. The proposal is refuted if relational scoring fails to shrink the mismatch degradation relative to the best baseline. It is refuted equally if the cheap global-rate and HuBERT ablations recover the same robustness.