Phone Boundaries as Measured Transitions in Self-Supervised Speech Space

Automatic phonetic labeling tools treat a phone boundary as an instant: one frame belongs to /a/, the next to /t/. Speech is not produced that way. The tongue and lips start moving toward the next sound before the current one ends. This overlap is called co-articulation. It is strongest exactly where aligners fail: on stops, nasals, and spontaneous speech. The Montreal Forced Aligner, run on spontaneous Cantonese, places only 23.5% of stop boundaries within 25 ms of a human annotator, while vowel onsets reach 93.84% (Performance of Montreal Forced Aligner on Cantonese Spontaneous Speech). We propose to model each boundary as a short, measurable transition. Self-supervised speech models are networks pre-trained on raw untranscribed audio. Inside such a model, we fit a smooth blend of frames moving from the left phone's typical representation to the right phone's. The midpoint of the blend is the boundary. Its width and its skew are a per-token measurement of co-articulation, and they double as a statement of how uncertain the boundary is.

This works because of a specific property of these models. In HuBERT and WavLM, phones sit in a roughly linear space organized by phonological features such as voicing and rounding. That structure transfers zero-shot from English to 95 other languages, with 94% and 92% analogy accuracy against 19% for MFCCs, the classic hand-crafted spectral features ([b]=[d]-[t]+[p]: Self-supervised Speech Models Discover Phonological Vector Arithmetic). If phones are points in a near-linear space, a co-articulated stretch of speech should look like a path between two points. A model built on that path needs very few language-specific parameters. It can therefore be fit to a new language from a phonemic transcript alone, with no manually placed boundaries in that language.

Core Claim

Our central claim is that a low-parameter transition model in frozen middle-layer representations does two things at once. First, it gives better phone boundaries than both triphone forced alignment and a supervised boundary detector, when moved to a language without boundary annotations. Second, its fitted transition parameters reproduce known co-articulation facts that were originally measured by hand-designed methods. Concretely, let $x_t$ be the frozen representation of frame $t$ inside a stretch that spans a left phone $L$ and a right phone $R$. We model

$$x_t = (1-\alpha_t)\,\mu_L + \alpha_t\,\mu_R + \varepsilon_t,\qquad \alpha_t = \sigma\!\left(\frac{t-b}{w_{\pm}}\right),$$

where $\mu_L$ and $\mu_R$ are phone centroids, $\varepsilon_t$ is Gaussian residual noise, $\sigma$ is the logistic function, $b$ is the boundary time, and $w_{\pm}$ is the transition width. The width takes separate values before and after $b$ ($w_-$ and $w_+$), which lets the model capture anticipatory and perseveratory spread separately. A segmental dynamic program searches over boundary times and widths for the whole utterance, given its phone sequence. The model outputs, for each boundary, a time, an uncertainty (from the width), and an asymmetry. Current aligners and boundary detectors give the time only.

Relation to Prior Work

Three papers motivate the gap. Leveraging Allophony in Self-Supervised Speech Models for Atypical Pronunciation Assessment fits 32-component Gaussian mixtures per phoneme over frozen WavLM and XLS-R features. It shows that these features split into subclusters according to neighboring context. However, it treats segments as static and depends on external forced alignment. We take its evidence that context lives in middle layers and model context as a path over time rather than as extra static clusters.

A Quantitative Measure of the Impact of Coarticulation on Phone Discriminability measures co-articulation as the loss in discriminability when a phone's context changes. It uses MFCCs, dynamic time warping, and segments placed by a forced aligner. It finds that anticipatory effects dominate, that fricatives resist context (under 5%) while stops and nasals do not (over 8–10%), and that English vowels and Japanese consonants are the more context-sensitive classes. These findings become our validation targets. If our fitted widths and skews are measuring co-articulation, they should recover these orderings on the same corpora, the Wall Street Journal corpus and the Corpus of Spontaneous Japanese, without any hand-crafted features.

Exploring Self-Supervised Speech Representations for Cross-lingual Acoustic-to-Articulatory Inversion recovers tongue and lip trajectories from audio. It shows that frozen multilingual representations carry articulator motion across languages: Dutch correlation 0.564 with MMS-1b versus 0.330 with MFCCs. It also shows that fine-tuning for speech recognition destroys this information. That result shapes our design: we never fine-tune the encoder. It also predicts a testable ablation, described below.

Two other papers supply the closest neighbors to our method.

  • AlloGraph. Differentiable Allophone Graphs for Language-Universal Speech Recognition maps universal phones to language-specific phonemes using trainable graphs, reducing zero-shot phone error from 94.0% to 80.2%. Its mappings ignore context, and it never measures timing. We address timing and context and borrow none of its machinery.
  • wav2vec 2.0 stop classification. Automatic classification of stop realisation with wav2vec2.0 fine-tunes wav2vec 2.0 to decide whether a stop has an audible burst. It reaches up to 94% accuracy on the Japanese corpus and shows that Japanese pre-training cuts English annotation needs to 500–1,000 tokens. That classifier works on pre-cut segments and outputs labels, not timestamps. Its transfer result supports our premise that stop acoustics carry over between languages.

Research Program

The work proceeds in three stages.

Stage 1: build the transition aligner on English. We pick the encoder layer on held-out TIMIT using English data only. We estimate centroids and residual variances from TIMIT's hand-placed boundaries. We then check two things:

  • whether a single logistic path explains boundary frames better than a hard switch between phones;
  • whether the residual stays near phone-independent along the path, which is what the linear-geometry picture predicts.

Stage 2: transfer to new languages. For a target language we assume audio plus phonemic transcripts, but no boundaries. Centroids are initialized in two ways:

  • for phones seen in the source data, from English or other source-language centroids;
  • for unseen phones, by phonological feature arithmetic, for example building an implosive or an unreleased stop from the relevant feature-difference vectors.

We then re-estimate centroids by alternating alignment and averaging on the target audio, an expectation-maximization loop with no labels. This is the step that addresses labeling scalability. Its cost is the same transcript-only input that the forced aligner already needs, minus the aligner's acoustic training in the target language.

Stage 3: validate the transition parameters as co-articulation measurements. We check whether widths and skews:

  • recover the anticipatory dominance and the stiffness ordering of phone classes from the discriminability study;
  • recover the English-versus-Japanese reversal between vowels and consonants;
  • predict where humans and machines disagree about boundaries.

The last check matters because width is meant to serve as a calibrated uncertainty that downstream tools can use. One example is pronunciation scoring, where The Impact of Forced-Alignment Errors on Automatic Pronunciation Evaluation shows that 53.4 ms alignment errors on cleft-palate speech leave the hand-measured score explaining only 72.8% of the automatic score's variance for nasals.

Baselines

Do-nothing reference. Boundaries are placed at uniform fractions of each word or utterance span. This sets the floor for every timing metric.

Montreal Forced Aligner. This triphone acoustic-model aligner with speaker adaptation (Montreal Forced Aligner: Trainable Text-Speech Alignment Using Kaldi) is the strongest widely used tool for exactly this job, and it already models left and right context. We run it in two modes: its pre-trained English model, and retrained on the same target-language audio and transcripts our method sees. Lexicons are converted to the same phone set. The reported reference points are:

Beating it means matching it within 1 ms on English (Buckeye) and clearly exceeding it on Cantonese stops and on the Japanese corpus, where neither system sees target boundaries.

CTC forced alignment from a multilingual speech recognizer. Connectionist temporal classification (CTC) is the training objective that lets a recognizer align text to audio without boundary labels. Here we fine-tune the same encoder for recognition on the source data and read boundaries off its alignment. No reported result found on Lacuna for CTC-based phone boundary accuracy on any of these benchmarks, so we run it ourselves with the same source data, the same layer budget, and the same tuning trials.

Supervised frame-level boundary detector. A small network on the same frozen features predicts "boundary here" for each frame. It is trained on English TIMIT and Buckeye boundaries and applied zero-shot to targets. This is the most direct competitor to our method's key ingredient.

Ablations. Two ablations isolate what is new:

Why not just…?

Why not just retrain the Montreal Forced Aligner on the target language? It does work in-language: retraining cut Phonsay error from 25.2 to 16.6 ms. On English we expect only parity. Its failures concentrate on stops and checked syllables in spontaneous speech (23.5% and 32.15%). Those are the places where a hard switch between triphone states fits a gradual transition worst. Our Cantonese and Japanese comparisons give the retrained aligner the same target audio and transcripts. If it still matches us on stops, the transition model adds nothing for timing, and the contribution narrows to the co-articulation measurements.

Why not just use CTC alignment from a multilingual recognizer? Recognition training pushes representations toward discrete categories. The inversion study found that it erases the articulator motion our method relies on (Exploring Self-Supervised Speech Representations for Cross-lingual Acoustic-to-Articulatory Inversion). That evidence concerns articulator trajectories, though, not timestamps, so it does not settle the question. The fine-tuned-feature ablation and the CTC baseline test it directly.

Why not just train a boundary detector on frozen features? This is the real threat. It uses the same representations and may transfer well. Our bet is that a detector learns English-specific cues and a single time per boundary. The transition model, with roughly two parameters per boundary, should transfer more robustly and also yield width and skew. If the detector matches us zero-shot on the Japanese corpus and Cantonese, we would reposition the work as a measurement tool rather than an aligner.

Proposed Experiments

In-domain timing. On English (TIMIT and Buckeye), we report:

  • mean absolute boundary difference;
  • the percentage of boundaries within 10, 20, and 25 ms;
  • results broken down by phone class.

Cross-lingual timing. Models built from English only are adapted with target transcripts and evaluated on two targets:

  • the hand-corrected Corpus of Spontaneous Japanese, which has manual segment boundaries;
  • the Cantonese spontaneous gold standard, with the stop-onset and checked-syllable offset breakdowns used in its original evaluation.

We will also try VoxAngeles for unseen-phone coverage, if its segment timings prove usable.

Co-articulation validity. We correlate fitted widths with the per-class coarticulation scores reported for the Wall Street Journal corpus and the Corpus of Spontaneous Japanese. We then test two predictions:

  • the anticipatory width $w_-$ exceeds the perseveratory width $w_+$ for most phone classes;
  • the vowel/consonant reversal between English and Japanese shows up without being built in.

Uncertainty. We test whether width predicts how far automatic boundaries fall from human ones, using rank correlation and calibration curves. If clinical data can be obtained, we also test whether weighting frames by $\alpha_t$ restores agreement between automatic and hand-computed pronunciation scores on atypical speech.

Decisive checks. Three results decide the claim. Stop-onset accuracy on Cantonese must clearly exceed 23.5%. The method must beat the frozen-feature boundary detector on both Japanese and Cantonese. The asymmetry and stiffness orderings must emerge without supervision. If the first two fail and the third succeeds, the method is a scalable co-articulation instrument rather than a better aligner.