Speech has no spaces between its sounds, so a system that cuts it into phones or phrases must decide where the cuts go. Duration is one of the strongest cues for those decisions. Listeners use the length of a single /s/ to tell Dutch "eens pot" from "een spot", and Japanese listeners separate short from long stop consonants. That cue only works in relative terms, though. A fast talker's long consonant can be shorter than a slow talker's short one. Phonetics research shows that listeners solve this by comparing a sound's length with its immediate neighbourhood, roughly the surrounding two syllables. Current automatic segmenters do not do this. They either ignore duration entirely or measure it in raw milliseconds or against dictionary pronunciations.
We propose a landmark clock: a running estimate of local speaking tempo built from acoustic landmarks. Landmarks are the sparse, physically well-defined instants where speech changes abruptly or peaks, such as vowel energy maxima, stop bursts, and the moments the vocal folds start or stop vibrating. The clock re-expresses every candidate segment's duration in units of nearby inter-landmark intervals. The same detector that finds anchors in the signal then supplies the rate normalization that duration cues need, with no forced alignment and no pronunciation dictionary. We test whether segmenters that judge duration on this clock place phone and phrase boundaries better than segmenters that use milliseconds or utterance-level rate. We expect the clearest gains in spontaneous speech, where tempo changes within a single sentence.
Core Claim
Duration priors in segmentation help only when they are relational and local. Landmark intervals give the right local unit cheaply and robustly. Concretely, adding a duration prior expressed in landmark-clock units to a strong unsupervised boundary detector should improve boundary accuracy on conversational speech (Buckeye) more than on read speech (TIMIT). It should also beat the same prior expressed in milliseconds or normalized by whole-utterance rate. The same clock should let a dictionary-free prosodic phrase-boundary detector close the gap to phone-aligned features.
Where This Sits in Prior Work
The motivating evidence comes from perception. Durational structure of Japanese single/geminate stops shows that absolute closure durations overlap across speaking rates. The ratio of closure to word duration, however, classifies singletons versus geminates at 97.5%, and a closure-to-subword ratio reaches 97.2%. The perception boundary between single and geminate stops finds that the surrounding disyllable is the best local anchor for the listener's category boundary ($R^2 = .990$). Segmenting ambiguous phrases using phoneme duration shows that, among the acoustic cues it tested, only /s/ duration drove segmentation. These papers establish that a local two-syllable window is the right normalizer, but they measure it from hand-labelled segments. Our clock is an automatic stand-in for that disyllabic anchor.
Two computational systems use duration but leave the normalizer external. Fine-Tracker raised correct lexical-embedding disambiguation from 10/28 to 18/28 stimuli by adding duration. Its per-phoneme temporal structure, however, is set by hand. Non-segmental duration feature extraction for prosodic classification is the closest existing work. It replaces phone alignment with energy peaks and valleys and computes rhyme and syllable durations from them. This matches segmental features for pitch accent (F 85.8 vs. 85.3) but trails them for phrase boundaries (77.9 vs. 80.1). Its rate normalization still divides by dictionary phone counts, and it was tested only on read news speech. We keep its peak/valley detector but replace the dictionary normalization with the local landmark clock. We also move the idea from prosodic classification to boundary decoding and to conversational speech.
The boundary detectors we build on supply the method. Self-Supervised Contrastive Learning for Unsupervised Phoneme Segmentation scores boundaries from raw waveforms using a convolutional network trained to tell adjacent frames from distant ones. It reaches an R-value of 86.02 on TIMIT and 79.69 on Buckeye, but picks peaks with a threshold that ignores elapsed time between candidates. Blind Phoneme Segmentation With Temporal Prediction Errors places boundaries at spikes of frame-prediction error and reaches R = 76.4. Automatic phonetic segmentation using boundary models shows that dedicated one-state transition models sharpen supervised alignment to 93.92% agreement within 20 ms. That matches the 93.49% agreement between human annotators.
The landmark side supplies the anchors and one constraint. Auto-Landmark releases TIMIT landmark annotations for five classes (glottal, burst, sonorant, voiced fricative, fricative) and detectors with Landmark Error Rate (LER) as low as 31.3%, against 56.5–56.7% for filter-based tools such as SpeechMark. It also reports that wav2vec 2.0 features hurt landmark detection, because their 10 ms stride blurs events closer together than that. This rules out building the clock on standard self-supervised frames. Instead, the clock is built mainly from syllable-scale peaks and valleys, which are tens to hundreds of milliseconds apart and are detected reliably (peak F-measure over 92% in the Dashiell et al. paper).
Research Program
The clock is computed as follows. Detected sonority peaks and inter-syllable energy valleys give a sequence of valley-to-valley intervals. At time $t$, the local tempo $\tau(t)$ is the median length of the $K$ intervals nearest $t$; the default is $K=2$, mirroring the disyllabic anchor, and $K$ is swept as an ablation. A candidate segment of raw length $d$ starting near $t$ gets the clock-normalized length $\tilde d = d / \tau(t)$. Burst and glottal landmarks from Auto-Landmark are used only as optional sub-syllabic anchors, so that errors in rare classes cannot corrupt the clock.
Boundary decoding then becomes a segmental search, replacing independent peak-picking. Given frame boundary scores $s(b)$ from a fixed detector, we choose boundaries $b_1 < \dots < b_n$ to maximize
$$\sum_{i} s(b_i) + \lambda \sum_i \log p_\theta\!\left(\frac{b_{i+1}-b_i}{\tau(b_i)}\right).$$
Here $p_\theta$ is a density over clock-normalized segment lengths and $\lambda$ weights the duration prior. The density is fit without labels on the detector's own confident segments, or with labels for the supervised variant. Exact dynamic programming over a bounded maximum segment length keeps decoding linear in utterance length.
For prosody, we recompute Dashiell et al.'s rhyme and syllable durations and divide them by $\tau$ in place of dictionary-expected durations. We then add the duration of the final inter-landmark interval before each valley. This targets the phrase-final lengthening that their valley features blurred.
The analysis question running through the program is where the gain comes from. Throughout, we compare the same decoder with three normalizers: raw milliseconds, whole-utterance landmark rate, and the local clock. We also report results binned by how much local tempo varies within each utterance.
Baselines
The naive reference is the contrastive detector of Kreuk et al. with its original threshold peak-picking. It is also the strongest existing unsupervised method in the file, with reported R-values of 86.02 on TIMIT and 79.69 on Buckeye. We retrain it from the released recipe on identical splits and use our reproduced numbers as the comparison. Every duration-aware variant consumes exactly these frame scores, so the only change is the decoder. Beating it means a gain in R-value and in 20 ms-tolerance F1 that exceeds the spread across at least five training seeds, on both corpora.
The prediction-error segmenter of Michel et al. (reported R = 76.4, F = 73.9% on TIMIT) serves as a second score source. It tests whether clock decoding helps a weaker, noisier detector more than a strong one.
The ablations that isolate the key ingredient are three decoders identical to ours except for normalization. The first uses a raw-millisecond duration prior. The second uses a fixed minimum peak separation, which is the common heuristic. The third uses a prior normalized by utterance-level landmark rate. Each gets the same budget for tuning $\lambda$ and the prior on the development set. A fourth control swaps the landmark-based $\tau$ for a tempo estimate from wav2vec 2.0 frame clustering. This tests whether landmarks specifically matter.
For phrase boundaries on the BU Radio News Corpus, the baselines are the reported phone-aligned segmental features (F = 80.1) and the dictionary-normalized non-segmental features (F = 77.9) from Dashiell et al.. Both are reimplemented with the same classifier and folds. Success means reaching or exceeding 80.1 without alignment or a dictionary.
Why not just…?
Why not just add a duration prior in milliseconds, or a minimum distance between peaks? This is the most serious threat, and it is cheap. On TIMIT read speech, tempo varies little within a sentence, and a millisecond prior may capture most of the gain. The evidence against it as a general solution is the production data from Hirata et al.: word durations range from 306 to 827 ms across rates, and absolute closure durations overlap between categories. The decisive test is Buckeye, binned by within-utterance tempo variance. If the millisecond prior matches the clock in the high-variance bins, the central claim fails and we will report it as a negative result.
Why not just use self-supervised features such as wav2vec 2.0 and let the model learn duration implicitly? Auto-Landmark found that these features degrade landmark detection because of their 10 ms stride. Kreuk et al. found that adding recurrent context hurt boundary placement, which suggests that implicit long-range modelling does not produce useful duration behaviour for this task. Self-supervised features still enter as a control: as a tempo source for $\tau$, and as an alternative frame-score source. If a self-supervised tempo estimate equals the landmark clock, landmarks are not essential to the idea and we will say so.
Why not just use forced alignment or a dictionary for rate normalization, as existing prosody systems do? It works on read English with a lexicon; the segmental baseline's 80.1 shows this. It is unavailable for low-resource languages, and it degrades on spontaneous speech, where Buckeye annotators agree only 77.8% of the time on vowel boundaries (Buckeye Corpus). The BU experiment measures how much accuracy the dictionary-free clock gives up, if any. A transfer check on German PHONDAT, where self-supervised augmentation lifted the contrastive model only from 55.34 to 75.58, tests whether the clock carries across languages without any lexicon.
Proposed Experiments
The first experiment is unsupervised phoneme segmentation on TIMIT and Buckeye. It uses R-value and F1 at a 20 ms tolerance and compares the threshold baseline, the millisecond, minimum-distance, and utterance-rate decoders, and the local clock, all on reproduced Kreuk et al. scores and on Michel et al. scores. The prediction is a small gain on TIMIT and a larger one on Buckeye. The second experiment is phrase-boundary and pitch-accent detection on BU Radio News, with clock-normalized features replacing dictionary normalization; here the target is the reported 80.1 phrase-boundary F. The third is a robustness check. We build the clock from the SpeechMark-style filter detector (LER about 56.5%) instead of the trained Auto-Landmark detector (31.3%), to measure how much clock quality depends on landmark accuracy. We also inject synthetic peak deletions to mimic reduction in conversational speech.
The decisive analysis is the curve of boundary F1 against within-utterance tempo variance on Buckeye. The claim holds only if the local clock's advantage over the millisecond and utterance-rate priors grows with that variance. A flat or reversed curve would mean that relational local duration, however important for listeners, adds nothing a well-tuned absolute prior cannot already supply to machine segmenters.
References
- Durational structure of Japanese single/geminate stops in three- and four-mora words spoken at varied rates
- The perception boundary between single and geminate stops in 3- and 4-mora Japanese words
- Segmenting ambiguous phrases using phoneme duration
- Using durational cues in a computational model of spoken-word recognition
- Non-segmental duration feature extraction for prosodic classification
- Self-Supervised Contrastive Learning for Unsupervised Phoneme Segmentation
- Blind Phoneme Segmentation With Temporal Prediction Errors
- Automatic phonetic segmentation using boundary models
- Auto-Landmark: Acoustic Landmark Dataset and Open-Source Toolkit for Landmark Extraction
- Speechmark: landmark detection tool for speech analysis
- The buckeye corpus of speech: updates and enhancements