Masked Duration Prediction: Learning Rate-Invariant Timing Priors for Speech Segmentation

Speech has no spaces between words. Listeners still separate "ham" from the start of "hamster," or a single Japanese stop from a doubled one, and they do it partly by timing. The relevant differences are often a few percent of a syllable, below what people can consciously hear. Speech recognizers have long modeled timing badly. A hidden Markov model (HMM) decides how long to stay in a state by repeatedly flipping the same biased coin, so it implies that the most likely duration of every sound is a single 10 ms frame. Fixes for this, such as explicit duration distributions and semi-Markov models (models that score whole segments with their lengths rather than frame-by-frame transitions), have used static per-unit distributions, absolute frame counts, or local left-to-right context. Perception research points to a different target. Listeners judge a duration relative to a nearby anchor, such as the surrounding disyllable, which cancels out speaking rate as it happens.

We propose a masked duration model. It is a bidirectional Transformer trained the way BERT-style masked language models are trained, except that the masked quantities are segment durations rather than words. Given a sequence of speech segments (their identities, their acoustic embeddings, and the durations of all but a few), it predicts the hidden durations. It predicts each one as a discrete, freely shaped distribution over its ratio to a local anchor, not as an absolute frame count. This model then supplies the duration term in a semi-Markov segmenter that runs on frozen self-supervised speech features. The claim is that relational targets plus two-sided context give a duration prior with three properties. It has the right shape: a non-zero mode and a right skew. It holds up when speaking rate shifts away from the training data. It is sensitive enough to separate embedded syllables from monosyllabic words, which neither static priors nor implicit duration learning in self-supervised encoders reliably do.

Core Claim and How It Differs

The question this work answers is whether duration should be treated as a masked, context-predicted, rate-relative quantity. Each existing line of work answers only part of that.

The duration-modeling papers supply the motivation and the competitors.

  • Using Duration Information in Cantonese Connected-Digit Recognition fits Gamma distributions to HMM state durations and scores them in a modified Viterbi search, the standard best-path decoding algorithm. Digit accuracy rises from 95.09% to 97.45% and insertions fall from 418 to 127. However, deletions rise slightly because the static prior penalizes fast, coarticulated digits. That deletion increase is exactly the failure a rate-relative prior should remove.
  • Applying word duration constraints by using unrolled HMMs encodes word duration histograms into the HMM topology and conditions on whether a word precedes a pause. It shows that context changes the right prior, but it does so through a state explosion that does not scale.
  • Phone Duration Modeling for LVCSR Using Neural Networks is the closest method. A feedforward network predicts a discrete distribution over phone durations from local phone labels and normalized preceding durations, and it is applied as post-hoc lattice rescoring. It gives 1% to 8% relative word error rate (WER) gains and beats log-normal models by 1% to 3%.

We differ from the last paper on three points. Our model uses full bidirectional context, including following durations rather than only preceding ones. Its target is a ratio to a local anchor rather than a frame count. Its prior is applied inside segmental decoding rather than only in rescoring. Hidden semi-Markov model based speech synthesis argues for that last choice: post-hoc duration application creates a train–decode mismatch, and joint semi-Markov training was preferred 60–80% of the time.

The perception work supplies the target parameterization.

These papers motivate a learned, relational, context-conditioned prior. Their overlapping distributions argue against hard thresholds.

The masked-modeling work supplies the training recipe and marks the open space.

That last finding constrains our design. We do not ask the frame encoder to learn timing. Boundaries stay local, and duration is modeled one level up, over segments.

The Model

Let segments $1..N$ have durations $d_i$ in frames. For segment $i$, the anchor $A_i$ is the summed duration of its neighbors within a window (for example, the adjacent syllable), excluding $i$ itself so the target is not leaked. The model predicts a categorical distribution over binned log-ratios $\rho_i = \log(d_i / A_i)$. Training masks a random 15–30% of durations, along with any anchor that would contain them, and minimizes

$$ \mathcal{L} = -\sum_{i \in M} \log p_\theta\left(\mathrm{bin}(\rho_i) \mid u_{1:N}, e_{1:N}, \{d_j\}_{j \notin M}\right). $$

Here $M$ is the masked set, $u_j$ are segment identities (phone labels, or discrete self-supervised units in the unsupervised setting), and $e_j$ are pooled HuBERT embeddings of each segment. Because the output is a free categorical distribution over bins, it can have a non-zero mode and a right skew, unlike the geometric distribution an HMM implies. To compare the model's scores against absolute-duration baselines, we convert ratio probabilities to frame probabilities, $p(d_i) = p(\rho_i)/d_i$. This Jacobian correction is applied once, so the comparison stays in the same units.

Using the Prior in Decoding

The segmenter scores a candidate segmentation as frame-level boundary evidence from a Strgar-style readout plus $\lambda \sum_i \log p(d_i \mid \text{context})$. Maximum segment length is capped at $D$ frames, which keeps semi-Markov Viterbi at $O(TD)$ cost for $T$ frames. This bound answers the decoding-cost objection raised against true semi-Markov search.

The prior depends on neighbors' durations, so exact decoding is not Markov. We use an iterated approximation instead. A first pass decodes without the duration term. Each later pass decodes again with the prior conditioned on the previous pass's neighbors. We expect convergence in two to three passes and will report it.

Research Program

Phase one trains the masked duration model on forced phone alignments from TIMIT and Buckeye. The model learns a relational prior, and we check whether its learned anchors reproduce the disyllable-like behavior found in perception studies.

Phase two removes labels. Segments come from the unsupervised segmenter, and segment identities come from clustered HuBERT units. Segmenting and re-training the duration model then alternate, so the system bootstraps without HMM forced alignments. Avoiding those alignments removes the dependency Phone Duration Modeling for LVCSR Using Neural Networks flags as vulnerable, since its alignments come from a geometric-biased HMM.

Phase three moves the same prior to the word level for the lexical-embedding question.

Baselines

The naive reference is the same frame-level boundary evidence decoded with no duration term, equivalent to peak picking or a geometric prior. It shows how much any duration modeling adds on top of strong self-supervised boundary evidence.

The strongest segmentation competitor is the layer-wise readout of Phoneme Segmentation Using Self-Supervised Speech Models. It reports TIMIT F1 96.60 supervised and 89.71 trained on noisy unsupervised labels. We run it on the same frozen HuBERT checkpoint and with the same label source. Our system adds only the duration term on top of it. Beating it means a statistically reliable F1 gain over 89.71 in the unsupervised regime on TIMIT, and any gain on Buckeye. For Buckeye F1 we found no reported result on Lacuna, so we re-run the readout ourselves. In the supervised regime, at 96.60, we expect little headroom and treat matching it as sufficient.

The strongest duration-model competitor is the feedforward network of Phone Duration Modeling for LVCSR Using Neural Networks. We reimplement it with (3,3) phone context, normalized preceding durations, and its score normalization. It is trained on identical alignments with the same hyperparameter search budget and plugged into the same decoder, both as a rescorer and inside decoding. Its reported effect is 1% to 8% relative WER gain over LF-MMI systems. We beat it if we achieve lower held-out duration negative log-likelihood and better segmentation and embedding accuracy, with the largest margins on rate-shifted test splits.

We also run the static Gamma prior of Using Duration Information in Cantonese Connected-Digit Recognition, fitted per unit. Its reported numbers on the digit corpus it was built for are 95.09% to 97.45% accuracy and 418 to 127 insertions. Our target there is to keep its insertion reduction without its deletion increase.

Three ablations isolate the ingredients:

  • an absolute-frame target in place of the ratio target, with everything else fixed;
  • left-only context in place of bidirectional masking;
  • post-hoc rescoring in place of in-decoder use.

Why not just…?

Why not just fine-tune the self-supervised readout and let it learn timing implicitly? This is a real threat on clean read speech, where the readout already reaches 96.60 F1. It is weaker on spontaneous speech and in label-poor settings. Self-Supervised Contrastive Learning for Unsupervised Phoneme Segmentation found that wider context hurt boundary precision, and the unsupervised readout falls to 89.71. The experiment that settles it is the Dutch embedding test described below, where we probe a fine-tuned readout directly. If the probe already separates embedded from monosyllabic syllables as well as our prior does, the explicit model is unnecessary and we will say so.

Why not just add a speaking-rate feature to the neural duration model of Hadian et al.? This is the most serious alternative, because a network that sees neighbors' durations could learn rate normalization on its own. The absolute-versus-ratio ablation is designed to answer it, with matched inputs on both sides. The test is a rate-shift split: train on each speaker's middle-rate utterances and test on the fastest and slowest quantiles. If the absolute model with a rate input matches the ratio model there, the relational target contributes nothing. The remaining contribution would then be bidirectionality and in-decoder use, and we would reframe the work around those.

Why not just time-stretch training audio to cover rate variation? Uniform stretching scales every segment equally. Real rate change is not uniform: Durational structure of Japanese single/geminate stops in three- and four-mora words spoken at varied rates shows word means moving from 827 ms to 306 ms while ratios stay stable. Stretched data therefore teaches the wrong invariance. We include a stretch-augmented absolute-target model as a baseline and evaluate it only on naturally rate-varied test speech.

Proposed Experiments

Segmentation. On TIMIT and Buckeye we report boundary F1 and R-value in both the supervised and unsupervised regimes, together with held-out duration negative log-likelihood. We also report the number of decoding passes and the runtime cost.

Rate shift. On Buckeye, using the rate-quantile split, we track how error rate and duration likelihood change from middle-rate to extreme-rate speech. On Aurora 2 connected digits, split by utterance rate and noise level, we report insertions and deletions separately. The unrolled-HMM paper reports gains there, but we found no numeric WER on Lacuna for Aurora 2 and will re-run that baseline ourselves.

Lexical embeddings. On word alignments from the Spoken Dutch Corpus, each test item is a syllable that is acoustically matched across both cases. We ask whether the prior's likelihood ratio classifies it as a monosyllabic word or the first syllable of a longer word. We test separately on read and on interview speech, where the reported shortening is about 2.5% and 9% respectively. The comparisons are a fine-tuned HuBERT probe, the Gamma prior and the Hadian-style network.

Decisive checks. The proposal succeeds if three things hold. The ratio model beats the absolute-plus-rate model on extreme-rate splits. Bidirectional masking beats left-only context on embedding classification. In-decoder use beats rescoring in the unsupervised regime. If all three fail, durations are already captured implicitly, and that negative result is itself worth reporting.