Forced alignment means finding when each phoneme in a known transcript starts and ends in the audio. It underlies phonetic research, audio editing and data preparation for speech synthesis. Neural aligners trained with connectionist temporal classification (CTC) are fast. CTC is a loss that lets a network learn frame-level label sequences from unaligned transcripts by summing over all possible alignments. Its weakness is that it collapses each phoneme into a single-frame spike and lets that spike drift in time. Less Peaky and More Accurate CTC Forced Alignment by Label Priors fixes much of the spikiness by penalizing the blank symbol. It also reports something awkward: a 5-million-parameter local network beats large Transformer and Conformer models at boundary placement, because big models use distant context to "guess" labels in ambiguous stretches. This sits uneasily with the fact that self-supervised speech models, such as wav2vec 2.0, HuBERT and WavLM, give the best phonetic features available. These are large Transformers pre-trained on unlabelled audio by predicting masked portions.
We propose a specific explanation for this tension, and an intervention that follows from it. Self-Supervised Speech Models Encode Phonetic Context via Position-dependent Orthogonal Subspaces showed that each 20 ms frame of these models stores three phones at once: the current phone, the previous phone and the next phone, each in its own mutually orthogonal linear subspace. A CTC output layer is rewarded for emitting a label at any frame where that label can be decoded. It therefore has every incentive to read the "next phone" subspace and fire before the phoneme actually begins, or to read the "previous phone" subspace and fire after it ends. Our claim is that this neighbor-phone content, rather than model size or global attention as such, is what smears and shifts boundaries when self-supervised features feed a CTC aligner. Projecting the neighbor subspaces out of the features before a small, strictly local CTC head should therefore give sharp boundaries while keeping the phonetic quality of the large model.
Core Claim
Let $h_t$ be the self-supervised representation at frame $t$. Let $U_{-1}$ and $U_{+1}$ be orthonormal bases for the previous-phone and next-phone subspaces, made orthogonal to the current-phone basis $U_0$. The aligner sees only
$$\tilde h_t = \left(I - U_{-1}U_{-1}^\top - U_{+1}U_{+1}^\top\right) h_t ,$$
where $I$ is the identity. A local CTC head is then trained on $\tilde h_t$ with the label-prior loss of Huang et al.
The hypothesis is that a frame should be able to emit phone $p$ only when $p$ is physically present at that frame. This is the local-sufficiency principle that Search for Information Bearing Components in Speech measured: phonetic information is concentrated in roughly a 200 ms window and decays steeply beyond about 100 ms. The projection removes the head's ability to drift inside the span where a neighbor phone is still decodable. Label priors alone cannot do this; they spread probability mass but do not restrict where it may go.
Three predictions follow. The projection should cut boundary error and emission offset relative to the same pipeline without it, at roughly unchanged phone error rate. The gain should be largest for WavLM, which Choi et al. found keeps the widest contextual window. With the projection in place, widening the head's receptive field should hurt much less than it does without the projection.
Research Program
The first stage is diagnostic. We train label-prior CTC heads on frozen layer-weighted features from wav2vec 2.0, HuBERT and WavLM, all loaded from TorchAudio 2.1. We follow Huang et al. and train on LibriSpeech 960h with lexicon-derived phone transcripts. Then, layer by layer, we measure how much of the trained head's weight energy falls inside $U_{-1}$ and $U_{+1}$, and correlate this with phoneme boundary error and with the signed offset of emissions from true phone centers.
Choi et al.'s work was purely a probing study. This diagnostic turns their geometry into a testable account of aligner failure. If neighbor-subspace energy does not predict boundary error, the hypothesis is wrong and we stop there.
The second stage is the intervention. We estimate $U_{-1}$, $U_0$ and $U_{+1}$ by fitting linear probes for the previous, current and next phone on TIMIT training frames with gold time alignments, then orthogonalizing the probes. This uses timed labels, which is a real cost, so we test two ways of reducing it. First, we check whether subspaces estimated on TIMIT transfer unchanged to conversational Buckeye. Second, we try a bootstrapped variant that estimates the subspaces from the aligner's own first-pass alignments on LibriSpeech and iterates. If the bootstrapped variant matches the TIMIT-estimated one, the method needs no timed supervision.
The third stage tests whether the projection can stand in for architectural locality. We vary the head's receptive field from a single frame up to about 31 frames, the optimum that Local Dense Synthesizer Attention found for recognition. We also compare against restricting the backbone's own attention window, in the style of On-device Streaming Discrete Speech Units. That work found representation quality plateaus quickly as windows widen but never checked whether the orthogonal context subspaces survive windowing.
Baselines
The naive reference point is standard CTC on the same frozen self-supervised features with the same local head. Huang et al. report that standard CTC predicts an average phoneme duration of 21 ms on Buckeye against roughly 82 ms in the human annotation, so we expect severe spikiness here.
The strongest existing method is the label-prior TDNN-FFN aligner of Huang et al. It is the strongest competitor because it is the published method that directly targets boundary accuracy and already beats large models. On Buckeye and TIMIT it reports a 12–40% reduction in phoneme and word boundary error relative to standard CTC, and an average predicted duration of 74 ms. The file reports no absolute phoneme boundary error value, so no reported result was found on Lacuna for absolute boundary error. We will therefore retrain it ourselves on LibriSpeech 960h with the same phone inventory, lexicon, label-prior exponent search and epoch budget as our models. Beating it means lower mean absolute boundary error and higher within-20 ms accuracy on both Buckeye and TIMIT, with duration closer to ground truth.
The Montreal Forced Aligner is the classical reference, a hidden-Markov-model pipeline with explicit duration modelling. We run it on the same transcripts and pronunciation dictionary. Huang et al. report it needed nearly 10 minutes on 8 CPUs for 20 hours of Buckeye, against under one minute on 4 GPUs for the neural aligner.
The obvious-combination baseline is the most important one: frozen WavLM features, the same local head and the label-prior loss, but no projection. It isolates the key ingredient. Our method beats it only if boundary error drops by more than seed variance at matched phone error rate.
Two controls close the ablation. One removes a random subspace of the same dimension as $U_{-1} \oplus U_{+1}$. The other removes the speaker subspace in the manner of Self-supervised Predictive Coding Models Encode Speaker and Phonetic Information in Orthogonal Subspaces. Gains must exceed both controls, which rules out generic dimensionality reduction or speaker normalization as the explanation.
We also report the fine-tuned wav2vec 2.0 aligner exposed by the TorchAudio forced-alignment API. For unlabelled boundary detection only, we report Self-Supervised Contrastive Learning for Unsupervised Phoneme Segmentation at its reported R-values of 86.02 on TIMIT and 79.69 on Buckeye. It is a ceiling check on boundary placement, not a competing aligner, because it assigns no phone identities. All neural models get identical training data, an identical hyperparameter-search budget and three seeds.
Why not just…?
Why not just put a small local head on frozen self-supervised features and call it done? This is the real threat. It is exactly the obvious-combination baseline above. If it matches the projected version, our mechanism is not needed. There is a reason to expect it to fall short. A head that sees only one frame still receives previous- and next-phone information, because Choi et al. show this information is packed into each single frame. Locality of the head therefore cannot remove it. The receptive-field sweep and the diagnostic correlation in stage one settle the question directly.
Why not just use lower layers, which are more acoustic and local? Lower layers carry more speaker and channel detail and less invariant phonetic content, as Exploring wav2vec 2.0 on Speaker Verification and Language Identification describes. We would expect worse phone recognition on conversational speech. We include single-layer heads at every depth in the diagnostic. If some low layer gives both good boundaries and acceptable phone error, we will report that as a simpler solution.
Why not just restrict the backbone's attention window? Windowing limits how far context can reach. It does not remove adjacent-phone content, which lies well within any useful window. Choi et al. report that quality drops sharply without some look-ahead, so the window cannot be shrunk enough to eliminate neighbor information without hurting features. We test windowed WavLM with and without projection, so the comparison is direct.
Why not just use the Montreal Forced Aligner? It avoids peakiness through explicit duration models, but it is slow, CPU-bound and cannot be trained end to end. If it remains more accurate on Buckeye, we report that honestly, along with the speed gap.
Proposed Experiments
We train on LibriSpeech 960h and evaluate boundary error, within-20 ms tolerance accuracy, predicted versus true duration, signed emission offset and phone error rate on the TIMIT test set and on Buckeye.
The decisive checks are four. First, neighbor-subspace energy in the head must correlate with boundary error across layers and models. Second, projection must beat the obvious-combination baseline and both random-subspace and speaker-subspace controls on Buckeye, using subspaces estimated only on TIMIT. Third, the bootstrapped, label-free subspace estimate must recover most of that gain. Fourth, with projection applied, boundary error should stay roughly flat as the head's receptive field widens from 1 to 31 frames, while without projection it should rise.
A failure on the first or second check falsifies the central claim. Success on all four would show that self-supervised models can serve as accurate aligners once their neighbor-phone content is removed, which resolves the conflict between Huang et al. and the probing literature.
References
- Less Peaky and More Accurate CTC Forced Alignment by Label Priors
- Self-Supervised Speech Models Encode Phonetic Context via Position-dependent Orthogonal Subspaces
- Search for Information Bearing Components in Speech
- TorchAudio 2.1: Advancing Speech Recognition, Self-Supervised Learning, and Audio Processing Components for Pytorch
- Transformer-based End-to-End Speech Recognition with Local Dense Synthesizer Attention
- On-device Streaming Discrete Speech Units
- Self-supervised Predictive Coding Models Encode Speaker and Phonetic Information in Orthogonal Subspaces
- Self-Supervised Contrastive Learning for Unsupervised Phoneme Segmentation
- Exploring wav2vec 2.0 on Speaker Verification and Language Identification