Can Forced Alignment Measure the Effect It Is Used to Study? Auditing Closed-Syllable Vowel Shortening in Running Speech

Across many languages, a vowel tends to be shorter when a consonant closes its syllable ("cat") than when the syllable ends in the vowel ("ca-"). Phoneticians call this closed-syllable compression, and it is usually a difference of tens of milliseconds. Large corpus studies of the effect now get their vowel durations from forced aligners, which are programs that take a recording and its transcript and place automatic boundaries between speech sounds. The most widely used aligner misplaces phone boundaries by about 17 to 25 ms on average. That error is roughly the same size as the effect being measured. The usual defence is that random errors average out over large samples. They do so only if the aligner errs the same way regardless of what follows the vowel, and there is reason to doubt that. A vowel running into /n/ or /l/ has a much less abrupt acoustic edge than a vowel running into /t/. A vowel at the end of an open syllable runs into the next word's onset.

This note proposes a measurement audit, together with a cheap correction. On speech that has manual phone boundaries, we will measure how aligner error at vowel onset and offset depends on syllable structure and coda type. We will then propagate that structure-dependent error into the regression coefficients that phoneticians actually report, and ask whether automatic measurement inflates, erases, or reverses the compression effect. The fix is to use a small manually annotated calibration subset to subtract the aligner's coefficient-level bias from estimates made on a large automatically aligned corpus. The core claim is about bias, not noise. If the error is correlated with syllable structure, more data makes the wrong answer more confident.

Where this sits in prior work

Three findings motivate the work. The Montreal Forced Aligner is built on Kaldi with triphone models and speaker adaptation. Its authors report a mean absolute phone boundary error of 17.0 ms on the conversational Buckeye corpus and 25.2 ms on laboratory Phonsay speech, falling to 16.6 ms after retraining. Those numbers are aggregates; the paper never breaks error down by syllable structure. Automatic measurement of vowel duration via structured prediction trains a boundary predictor to directly minimise millisecond error. It cuts vowel onset error to 5.21 ms against 16.67 ms for an HMM aligner, and it shows that regressions run on its output recover the same significant effects as manual measurement. That is the closest existing work. It differs from ours in two ways. It is restricted to isolated CVC words, so the open-versus-closed contrast never appears. It also checks only that the same coefficients come out significant, not whether they come out unbiased.

The third finding is Predicting vowel duration in spontaneous Canadian French speech. It found syllable structure non-significant for vowel duration (a −0.11 loading) next to intrinsic vowel properties (56.7% of variance in reading), using 4 speakers. That null result contradicts the laboratory consensus from studies such as Hajek and Stevens on Italian and Isei-Jaakkola and Asakawa on Japanese, Finnish and Czech. We do not take a side in that dispute. We ask a prior question: could measurement error that depends on syllable structure produce a null like that, or hide one?

The problem has a wider reach. Liberman's account of corpus phonetics argues that phonetic claims must be validated on large corpora. In speech synthesis, FastSpeech 2 trains its duration predictor on aligner-extracted targets, and SoftSpeech argues that aligner errors degrade synthesis. Together they imply that any structure-dependent aligner bias is inherited both by phonetic science and by the synthetic voices trained on it. The general concern, that aggregate scores hide whether specific linguistic behaviour is right, echoes the position paper on responsible text-to-speech evaluation.

Core Claim

Forced-aligner error at vowel boundaries is systematically different for open syllables, for syllables closed by sonorants, and for syllables closed by obstruents. Because of this, the closed-syllable compression coefficient estimated from automatic durations is biased, and the bias does not shrink as the corpus grows. The bias can be estimated and removed using a calibration subset of a few hundred manually segmented tokens per condition. The corrected estimate should match the manual-label estimate at a fraction of the annotation cost.

Research Program

The work has three stages.

Error decomposition. We take Buckeye, which has 20.7 hours of conversational speech from 40 speakers with manual phone boundaries, and Phonsay, which has 48 minutes of laboratory speech from 45 speakers. We syllabify both with a standard maximal-onset rule. Each vowel token is labelled as open, closed by a sonorant coda, or closed by an obstruent coda. We also record stress and position in the word, because the Canadian French study shows these confound structure. We run the aligners and compute signed onset error and signed offset error separately for each structure class. Signed error matters here because absolute error cannot reveal bias. The prediction is that offset error before sonorant codas is shifted in one direction relative to obstruent codas. This would mirror the later alignment of pitch targets before sonorant codas reported by Welby and Loevenbruck and Jilka and Möbius, which suggests that the vowel-to-sonorant edge is genuinely ambiguous.

Coefficient-level propagation. We fit the same mixed-effects model of vowel duration twice, once on manual durations and once on automatic durations. The model has fixed effects for structure class, vowel identity, stress and position, and random effects for speaker and word. We report the difference in the compression coefficient. To separate bias from noise, we subsample the automatic data at increasing sizes. If the automatic coefficient converges to a value outside the manual confidence interval, the error is bias. We also run a simulation in which structure-conditional error distributions estimated from Buckeye are added to the manual durations. The simulation asks how often an effect of a given true size flips sign or loses significance at the sample size of the Canadian French study (4 speakers). That tells us whether a measurement artefact could plausibly explain its null.

Calibration correction. Let $\hat\beta_{\text{auto}}^{N}$ be the compression coefficient estimated from automatic durations on the full corpus of $N$ tokens. Let $\hat\beta_{\text{auto}}^{n}$ and $\hat\beta_{\text{man}}^{n}$ be the coefficients from automatic and manual durations on a manually annotated subset of $n$ tokens. The corrected estimate is

$$\hat\beta_{\text{corr}} = \hat\beta_{\text{auto}}^{N} - \left(\hat\beta_{\text{auto}}^{n} - \hat\beta_{\text{man}}^{n}\right).$$

The bracketed term is the aligner's bias on the coefficient, measured where we have ground truth. We bootstrap over speakers in both sets to get confidence intervals. The question is how small $n$ can be while the corrected intervals still cover the manual estimate. Once that is known, we apply the procedure to LibriTTS, which has 585 hours from 2,456 speakers but no manual boundaries. We draw the calibration subset from Buckeye, which means testing how well a correction transfers across corpora. That transfer is an empirical question, and we will report it as one.

Baselines

The naive reference point is current practice: align with the off-the-shelf pretrained Montreal Forced Aligner, fit the regression on all tokens, and report the coefficient as is. Its reported mean phone boundary error is 17.0 ms on Buckeye and 25.2 ms on Phonsay (MFA). No reported result was found on Lacuna for its bias on a syllable-compression coefficient; producing that number is the point of stage two.

The strongest aligner baseline is the Montreal Forced Aligner retrained on in-domain data, which reached 16.6 ms on Phonsay. Retraining is the remedy practitioners would reach for first. We retrain on the same Buckeye training speakers that our calibration subset comes from, so it sees exactly the manual labels our correction sees.

The strongest specialised baseline is the direct-loss structured predictor of Adi et al.. Its reported vowel onset deviation is 5.21 ms on Heller and Goldrick, against 16.67 ms for an HMM aligner, and 9.84 ms under dataset mismatch against 35.90 ms. Because it was built for isolated CVC words, we reimplement it on windows centred on aligner-located vowels in Buckeye. We train it on the same calibration tokens and use the same tuning budget as the retrained aligner. Its error on open-syllable vowels is a genuine unknown.

A further baseline is the manual-subset-only estimate, $\hat\beta_{\text{man}}^{n}$ on its own, which is unbiased by construction.

The ablation that isolates our key ingredient is a correction that applies a single global boundary offset to all vowels instead of estimating bias at the coefficient level conditioned on structure.

Beating the baselines means the following. The corrected estimate must cover the manual full-corpus coefficient at nominal rates, which none of the uncorrected aligner variants should do if the bias hypothesis holds. Its intervals must also be clearly narrower than those of the manual-only estimate at the same $n$.

Why not just…?

Why not just use more data, since boundary errors average out? Averaging removes noise, not bias. The subsampling experiment in stage two tests this directly. If the automatic coefficient converges to the manual one as $N$ grows, this objection wins, and the proposal reduces to a useful null result that certifies current practice for this effect. We consider that a publishable outcome, not a failure.

Why not just retrain the aligner or use the more accurate structured predictor? Lower average error does not imply lower structure-dependent bias. A model can cut mean error and still systematically misplace vowel-to-sonorant edges. Both are run as baselines on equal data. If either one removes the coefficient bias without any correction, it is the better practical recommendation, and we will say so. The structured predictor is the most serious threat here, given its 5.21 ms onset error.

Why not just hand-annotate a subset and analyse only that? That estimate is unbiased, and if it is precise enough it makes the large corpus unnecessary. The comparison of interval widths at equal $n$ settles this. The correction is worth having only if it buys substantially more precision than the manual subset alone, especially for rarer cells such as unstressed vowels before sonorant codas.

Why not just add an open/closed flag to the model and trust it? A flag tells the model about structure. It does nothing about errors in the durations being modelled, and the Canadian French factor analysis shows that categorical structure variables can come out non-significant once other factors enter the model. Our audit asks whether such conclusions are measurement-driven.

Decisive Checks

The first check is whether signed offset error differs between sonorant-closed and obstruent-closed vowels in Buckeye, with a speaker-bootstrapped interval that excludes zero. If it does not, the bias mechanism is absent. The second is whether the naive automatic compression coefficient falls outside the manual 95% interval at full Buckeye scale. The third is whether the structure-conditional correction restores coverage while the global-offset ablation does not, which would show that conditioning on structure is the active ingredient. The fourth is how often the simulated measurement error, at a four-speaker sample size, turns a true compression effect into a non-significant one. Finally, on LibriTTS, we compare the corrected and naive compression estimates. A large gap would warn every synthesis system whose duration targets, like FastSpeech 2's, come from the same aligner.