Speech has no spaces between words. Listeners nonetheless find word edges, and part of how they do it is timing. A sound is slightly shorter when it sits inside a longer word than when it stands alone. A vowel stretches before a boundary. Pauses fall between phrases. Systems that try to discover words in unwritten languages directly from audio mostly discard this timing. They first turn the audio into a sequence of discrete "pseudo-phone" symbols. Then they either collapse repeated symbols or merge frequent symbol pairs, and both steps throw away how long each sound lasted. This proposal keeps those durations. It turns them into a soft, self-calibrating vote on where word boundaries fall, and tests whether that vote improves unsupervised word segmentation on Mboshi, an unwritten Bantu language.
The concrete bet is this. When repeated discrete units are compressed, record how long each run lasted and express that length relative to its local surroundings, which cancels out speaking rate. Then let a small timing model, re-fit from the segmenter's own guesses, multiply into the probability of placing a boundary. The work is new if it succeeds on two counts: it beats plain compression that keeps no durations, and it beats the earlier approach of inserting prosodic boundaries as hard, unbreakable markers.
Core Claim
In zero-resource word segmentation, the discrete-unit pipeline loses useful boundary information at the compression step. Recovering that information as a rate-normalized, probabilistic duration term inside the segmenter's sampler should raise boundary F-score on Mboshi. The gain should come from the duration values themselves, not merely from shorter sequences. The gain should also be largest when the corpus is smallest, because that is when the segmenter's word-frequency statistics are weakest.
Why the gap exists
The diagnosis that motivates this proposal comes from Unsupervised Word Segmentation from Discrete Speech Units in Low-Resource Settings. That paper feeds Bayesian and neural speech discretizations into word segmenters on the Mboshi-French corpus. The best result is a 64.7% boundary F-score, from Bayesian subspace HMM units. Neural VQ-VAE units top out at 52.7%. The authors attribute the gap to the long, redundant sequences that neural units produce at fixed frame rates. On the compression side, Exploring the Benefits of Tokenization of Discrete Acoustic Units shows that byte-pair merging of acoustic units fixes redundancy for sequence modeling. That merging erases frame counts. Neither paper asks what the erased durations were worth for segmentation.
The perception literature says they are worth something, but not much individually. Segmenting ambiguous phrases using phoneme duration finds that /s/ duration was the only measured acoustic property that tracked listeners' parsing of Dutch eens pot versus een spot. Durational information in word-initial lexical embeddings in spoken Dutch finds embedded syllables about 2.5% shorter than matching monosyllabic words in read speech and about 9% shorter in interviews. Those effects are real but sit inside heavily overlapping distributions. Two papers on Japanese consonant length supply the normalization idea we reuse: listeners judge a duration relative to a local window rather than in absolute milliseconds. Durational structure of Japanese single/geminate stops classifies single versus doubled stops at 97.5% accuracy using closure-to-word ratios. The perception boundary between single and geminate stops shows the perceptual category boundary tracks the surrounding disyllable with $R^2 = .990$.
Two findings constrain the design. Prosodic cues for syntactically-motivated junctures finds pauses at 97% of Cantonese syntactic boundaries, but pre-boundary lengthening at only 64%, with an average lengthening of 1.8% and a per-speaker range of 27% to 91%. Duration cues are therefore weak and speaker-dependent, so they must be soft and weighted. Using Duration Information in Cantonese Connected-Digit Recognition shows that Gamma-distributed duration models cut insertion errors from 418 to 127 and lifted digit accuracy from 95.09% to 97.45%. The same rigid penalties raised deletions in fast speech, which argues for a tunable weight rather than a fixed rule.
The closest prior work is Prosodic boundary information helps unsupervised word segmentation. It adds pause, vowel-length, onset-distance and pitch-reset cues to a grammar-based word segmenter and lifts spontaneous Japanese token F-score from about 44% to 55%. It has three limitations that matter here. It assumes oracle phone or syllable segmentation, it inserts boundaries as hard markers that the model cannot undo, and it reports that false positives are fatal, so it must trade recall away for precision. It was also tested only on English and Japanese. Our proposal starts from discovered units, uses a soft term the sampler can override, and targets an unwritten language.
Research Program
Start from a discrete unit sequence for each utterance. We use two substrates so the result does not depend on one encoder. The first is a VQ-VAE trained on the Mboshi audio itself, as in the Algayres study. The second is the Bayesian H-SHMM units from that same study, which also have durations.
Compress consecutive identical units into runs. Each run $i$ keeps its unit label $u_i$, its duration $d_i$ in frames, and the silence $s_i$ that follows it. The temporal feature at the potential boundary after run $i$ has two parts. The first is the pause $s_i$. The second is a relational length $r_i = \log\big(d_i / \bar d_{W_i}\big)$, where $\bar d_{W_i}$ is the mean run duration in a local window $W_i$ around run $i$. Following the disyllable result, the default window spans the two nearest syllable-like spans, found by an unsupervised energy-envelope nucleus detector. A fixed span of $k$ neighboring runs serves as a comparison.
The segmenter is a Gibbs-sampled Dirichlet-process word segmenter of the dpseg type, which Preliminary Experiments on Unsupervised Word Discovery in Mboshi found strongest on Mboshi phonemes, at 67.5% F1. Each sampling sweep decides, for every gap between runs, whether to place a word boundary there. We let timing enter that decision as one extra factor:
$$ p(b_i \mid b_{-i}, u, x) \;\propto\; p_{\text{seg}}(b_i \mid b_{-i}, u)\; p_\theta(x_i \mid b_i)^{\lambda} $$
The symbols are as follows. $b_i \in \{0,1\}$ marks whether there is a boundary after run $i$, and $b_{-i}$ is every other boundary decision. $p_{\text{seg}}$ is the usual lexicon-based probability from the segmenter. $x_i = (s_i, r_i)$ is the timing feature just defined. $p_\theta$ is a two-class timing model, boundary versus non-boundary, with Gamma-distributed pause and relational-length terms in the spirit of Zhu's duration models. $\lambda$ sets how much the timing vote counts.
No labels are used. Utterance ends are free positive examples and utterance-internal gaps are presumed mostly negative, so $\theta$ is initialized by contrasting those two sets. After each sweep, $\theta$ is re-fit from the sampler's current boundaries. The timing model and the lexicon therefore teach each other, much as alternating steps do in expectation-maximization.
Two features separate this from Ludusan et al. First, a strong lexical preference can outvote a misleading duration, so false positives are no longer fatal. Second, $\theta$ adapts to Mboshi's speakers and rhythm instead of using hand-set thresholds. The same multiplicative term can be attached to the neural bilingual segmenter used for the 64.7% result, so the comparison with the strongest system is like for like.
Baselines
The naive reference point is the segmenter run on uncompressed VQ-VAE frames, which reproduces the setting where the Algayres study reports 52.7% boundary F-score. A second floor is placing boundaries only at detected pauses, which tests whether "timing" just means "silence."
The strongest existing method is H-SHMM units fed to the neural bilingual segmenter, with a reported boundary F-score of 64.7% on MB-FR (Algayres et al.). It is the strongest competitor because it already uses cross-lingual acoustic priors and French translations, and because its Bayesian units implicitly absorb some duration through their transition structure. We will rerun it with released or re-implemented components on the identical train/test split. Our timing term will be added to that same pipeline, so the comparison isolates the timing factor rather than a change of segmenter. For context, the reverse-attention system of Unsupervised Word Segmentation from Speech with Attention reports 50.0%, and its oracle-phone topline is 61.0%.
The prosodic-prior baseline is Ludusan-style hard boundary insertion. It uses the same automatically extracted pause and relational-length features, thresholded for high precision as that paper recommends, and fed as fixed markers into the same dpseg sampler. A byte-pair-merged unit baseline follows Dekel & Castro Fernandez.
The ablation that isolates the key ingredient is run-length compression with durations discarded. It has identical sequences, identical sampler, and $\lambda = 0$. A second ablation keeps the duration feature but shuffles $d_i$ across runs of the same unit label within the corpus. This preserves every marginal statistic while destroying the alignment between a duration and its position. A third ablation uses absolute instead of relational durations.
Fairness rules apply to every system. All systems get the same sampler iteration count and five random seeds. Hyperparameters are tuned on the same 100-utterance development slice with gold boundaries: $\lambda$ for ours, the threshold for Ludusan-style insertion, the merge count for byte-pair merging, and the segmenter concentration for all. Nothing else in training sees gold labels.
Beating the compression-only ablation by more than two seed standard deviations, with the shuffled control falling back to that ablation's level, would show that durations carry boundary information. Matching or exceeding 64.7% with a monolingual segmenter on VQ-VAE units would show that timing can substitute for bilingual supervision. Improving the bilingual H-SHMM pipeline itself would be the strongest result.
Why not just…?
Why not just collapse repeated units and stop there? This is the real threat. The Algayres diagnosis was about sequence length, and silence removal alone already helped neural units. Compression may recover most of the 12-point gap between neural and Bayesian units, and the perceptual effects in Dutch are small, at 2.5% to 9%. The compression-only and shuffled-duration ablations exist to settle this. If they match the full model, the honest conclusion is that duration is not worth keeping for this task, and we will report it as such.
Why not just insert pauses as hard boundaries, as Ludusan et al. did? Pauses are the strongest single cue in Chow's data, so this could capture most of the benefit. Ludusan et al. themselves show that hard markers force a precision-for-recall trade. Lengthening is present at only 64% of boundaries and varies across speakers from 27% to 91%, which makes a fixed threshold a poor fit. The direct comparison against hard insertion on identical features, together with the pause-only floor, tells us whether the soft adaptive term adds anything beyond silences.
Why not just apply byte-pair merging to the units? Merging solves redundancy for sequence modeling. However, it groups units by corpus frequency without regard to syllable or word edges, and it discards frame counts. Merged tokens can also span a true word boundary, which caps achievable recall. The byte-pair baseline measures this directly. We will additionally report the fraction of merged tokens that straddle gold boundaries.
Why not just use a large pretrained multilingual speech encoder? Spoken-Term Discovery using Discrete Speech Units finds that HuBERT units work best when pretraining matches the evaluation language, which is impossible for Mboshi. Elmakies et al. note that such speech language models lack prosodic and duration information. A stronger encoder would therefore be complementary rather than a substitute. We include mismatched-language HuBERT units as a third substrate to check this.
Proposed Experiments and Decisive Checks
The primary benchmark is MB-FR/Mboshi5k, scored by boundary F-score, precision and recall, plus vocabulary retrieval, where the attention system reaches only 13.5%. To check that results transfer beyond one language, we run the monolingual variant on the ZeroSpeech 2017 languages of The zero resource speech challenge 2017, including surprise-language Wolof. There we report boundary F-score and the normalized edit distance used by DUSTED. We also run it on CSJ, where Ludusan et al. report 44% to 55% token F-score with phone input, to show what is lost when starting from discovered rather than oracle units.
Four analyses decide whether the idea holds. The first is the shuffled-duration control described above. The second is a data-scaling curve that subsamples Mboshi from 500 sentences to the full corpus. The claim predicts that the gap between the full model and the compression ablation widens as data shrinks; if it does not, the low-resource motivation fails. The third is a speaking-rate split comparing relational and absolute durations on the fastest and slowest utterance quintiles. Relational features should degrade less, as the Japanese results suggest. The fourth is an error breakdown by word length. Short, monosyllabic words are where Zhu's digits and Fine-Tracker's embeddings (Using durational cues in a computational model of spoken-word recognition, 10 to 18 of 28 stimuli) show duration mattering most, so gains should concentrate there. A deletion analysis on fast speech checks whether the learned $\lambda$ avoids the over-penalization Zhu observed.
One specific risk is that the timing model, initialized from utterance ends, learns utterance-final lengthening, which is stronger than word-final lengthening. That would make it over-propose boundaries at phrase edges. The per-position precision analysis will show whether this happens, and initializing from pause positions only is the fallback.
References
- Unsupervised Word Segmentation from Discrete Speech Units in Low-Resource Settings
- Exploring the Benefits of Tokenization of Discrete Acoustic Units
- Segmenting ambiguous phrases using phoneme duration
- Durational information in word-initial lexical embeddings in spoken Dutch
- Durational structure of Japanese single/geminate stops in three- and four-mora words spoken at varied rates
- The perception boundary between single and geminate stops in 3- and 4-mora Japanese words
- Prosodic cues for syntactically-motivated junctures
- Using Duration Information in Cantonese Connected-Digit Recognition
- Prosodic boundary information helps unsupervised word segmentation
- Preliminary Experiments on Unsupervised Word Discovery in Mboshi
- Unsupervised Word Segmentation from Speech with Attention
- Spoken-Term Discovery using Discrete Speech Units
- Unsupervised Speech Segmentation: A General Approach Using Speech Language Models
- The Zero Resource Speech Challenge 2017
- Using durational cues in a computational model of spoken-word recognition