Systems that transcribe stuttered or aphasic speech phoneme by phoneme rely on a large pretrained speech encoder that outputs, for every 20 ms frame, a probability over phonemes. A decoding graph then searches those probabilities for repetitions, skips and prolongations. The encoder is usually frozen, because the search step cannot pass gradients back to it, and everyone knows this is the weak link: if the encoder blurs a repeated "st-st" into a single phone, no graph can recover it. Fine-tuning the encoder with ordinary transcription losses is worse, because those losses reward the encoder for reproducing the fluent reference text. That is exactly how Whisper-style models learn to erase disfluencies.
We propose to adapt the encoder with low-rank adapters, which are small trainable matrices added beside frozen weights. The adapters are trained against the sum of probabilities over every path through a disfluency-aware graph, given only the text the speaker intended to say. This summed likelihood is differentiable even though the best-path search is not. The adapters can therefore learn to sharpen evidence for repeats, prolongations and omissions without being pushed to hide them. The training needs no verbatim clinical labels, only audio paired with the prompt the patient was reading. That is the kind of data clinics actually have.
Core Claim
Training an encoder against a fluent-text target teaches it to suppress disfluencies. Training it against a graph that explicitly permits disfluencies, through a low-rank update, teaches it to expose them. Write $x$ for the audio, $y$ for the intended phoneme sequence, and $G(y)$ for the acceptor built from $y$ with return arcs (repetitions), skip arcs (deletions) and self-loops (prolongations), each carrying a penalty weight $w(\pi)$. The encoder's frozen weights are $W_0$, and its low-rank adapter is $BA$. The objective is
$$\mathcal{L}(B,A) = -\log \sum_{\pi \in G(y)} e^{-w(\pi)} \prod_{t} p(\pi_t \mid x;\, W_0 + BA),$$
where $\pi$ ranges over frame-level paths through the graph and $p(\pi_t \mid x)$ is the encoder's phoneme probability at frame $t$. The forward algorithm computes this sum exactly and differentiably. At test time we keep the zero-shot best-path decoder of Dysfluent WFST unchanged and swap in the adapted emissions. The claim is new because that paper treats graph non-differentiability as a barrier to tuning the encoder at all. Here only the decoding step is non-differentiable, and the training signal comes from the graph's marginal likelihood.
The low-rank constraint is part of the method, not just a convenience. The graph is permissive, so a fully trainable encoder could reach high likelihood by drifting, for example by using loops to absorb noise frames. Restricting the update to rank $r$ limits how far phone identity can move from the pretrained solution. The evidence for this kind of regularization comes from other domains. In vision, Kong et al. found that LoRA on a frozen ViT avoided overfitting to synthetic forgery artifacts and raised cross-dataset AUC by 11.19% on average. In language models, On the Calibration of Large Language Models and Alignment found that low-rank tuning preserves calibration where full tuning on synthetic data collapses it. Whether either transfers to speech emissions is something we test, not something we assume.
Research Program
The method draws on three lines of prior work. The graph topology comes from two zero-shot or weakly supervised aligners. Kouzelis et al. built a modified acceptor with loop, skip and part-word arcs over a frozen Wav2Vec2 model, plus a mismatch estimate (Oracle Error Rate) that sets how flexible the graph is. Dysfluent WFST added prolongation self-loops and Gaussian-weighted arcs over frozen WavLM and wav2vec 2.0. Both keep the encoder fixed, and the second explicitly attributes its residual errors to noisy emissions and ambiguous insertion-versus-deletion paths. We reuse their graphs and target exactly that weakness. The adapter design comes from speech PEFT work. S2-LoRA found that value and output projections carry most of the needed change when adapting Whisper to child speech, with as little as 0.02% of parameters trained. LoRA-Whisper used separate adapters per language to avoid interference. Both trained toward standard transcripts, so they adapt the encoder in the direction that erases disfluency. The motivation comes from the gap between simulated and clinical data in learned dysfluency systems. SSDM drops from 90.0% F1 on VCTK++ to 69.9% on nfvPPA. UDM reaches 64.5 F1 on clinical phonetic dysfluency against 90.6 for human experts. Both train sizeable modules on simulated data and never adapt on reference-only clinical audio.
The work proceeds in three stages. First, we implement the graph-marginal loss over WavLM-Large emissions with adapters on attention value and output projections in all layers. We sweep rank over 4, 16 and 64, and we also try the shared-basis variant from S2-LoRA. Second, we train in two phases. Phase A uses simulated corpora (Libri-Dys, VCTK++, and the Claude+VITS simu-rep/del/ins sets) with fluent reference text only, deliberately discarding their verbatim labels. Phase B adapts further on clinical audio paired with reading prompts, using held-out speakers. Arc penalties start at the Dysfluent WFST settings and are tied to the per-utterance mismatch estimate of Kouzelis et al., so that fluent utterances are not pushed toward spurious loops. Third, we analyse what the adapters change. We compare per-layer update norms and singular-value spectra across ranks, and we measure the frame-level entropy and calibration error of emissions on prolonged and repeated segments compared with fluent ones. Together these test whether the adapters work by sharpening repeated-phone evidence or by something less benign.
Baselines
The naive reference point is the frozen encoder with the unchanged Dysfluent WFST decoder, run exactly as published. On simulated repetitions it reports 4.03% WPER (weighted phonetic error rate), against 14.23% for greedy decoding, and 100% repetition-count accuracy, versus 51% for YOLO-Stutter (Dysfluent WFST). The authors evaluated on 35 nfvPPA patients, but no reported result was found on Lacuna for that system's clinical numbers. We will therefore rerun it on our clinical splits. Because simu-rep is near ceiling, beating this baseline means a clear gain on simu-del and simu-ins and on clinical data, with no regression on simu-rep beyond noise.
The strongest supervised competitors are the learned dysfluency systems. They are H-UDM, which reports 73.1% word-level detection F1 on aphasia/dyslexia speech against 64.0% for Whisper-X (Towards Hierarchical Spoken Language Disfluency Modeling), and SSDM at 69.9% F1 on nfvPPA. For forced alignment of stuttered speech, the comparison point is Kouzelis et al., with 0.67 frame Overlap on DisfluenTIMIT against 0.50 for the Montreal Forced Aligner. Where code or splits allow, we evaluate on the same test speakers and metrics. Otherwise we compare against the published figures and say so explicitly. These systems use verbatim labels on simulated data, while ours does not. Matching them on clinical F1 would therefore already be meaningful, and exceeding H-UDM's 73.1 on its aphasia/dyslexia set would count as a clear win.
Two baselines isolate the key ingredient. The first trains the same adapters, at the same rank, data and step budget, with ordinary phoneme CTC against the fluent reference. This tests the claim that the graph, not adaptation itself, prevents disfluency erasure. Our prediction is that this baseline lowers WPER on fluent stretches but lowers repetition recall. The second trains the full encoder with the same graph-marginal loss and the same tuning budget (learning-rate sweep of equal size). This tests whether the low-rank constraint matters. A third, weaker baseline trains only a linear output layer on frozen features with the graph loss. It checks whether a cheap change in the output space captures most of the benefit.
Why not just…?
Why not just fine-tune with verbatim phoneme labels from the simulated corpora? This is the most serious threat. Libri-Dys provides nearly 4,000 hours with exact dysfluent transcripts, and supervised CTC on those labels may beat our weak signal on simulated test sets. We will include it, both LoRA and full fine-tuning, under the same budget. The question it cannot answer is clinical adaptation, because clinics rarely have verbatim phonemic transcripts, while our loss can keep adapting on prompt-paired patient audio. If verbatim-label training also wins on clinical held-out speakers after our Phase B adaptation, we will report that the weak-supervision angle adds nothing. The proposal would then reduce to a cheaper labeling-free alternative and not a better one.
Why not just fine-tune the whole encoder with the graph loss? Full tuning has more capacity, and nothing in the speech literature in this file proves low rank helps here. The vision and calibration results are indirect. The full-tuning ablation settles it on two axes: clinical F1 and boundary error, and fluent-speech phone error on held-out LibriSpeech-style audio as a forgetting probe. If full tuning matches on both, the adapters are only a storage convenience and we will say so.
Why not just keep the encoder frozen and improve the graph, for example with better arc weights or the mismatch-adaptive flexibility of Kouzelis et al.? The graph-only route has no training cost, and its authors identify emission noise, not topology, as the remaining error source. Our frozen-encoder baseline already includes the adaptive flexibility. If a tuned frozen-encoder graph closes most of the clinical gap, encoder adaptation is unnecessary.
Why not just clean up transcripts with a language model? Text-level methods such as DisAug work on word-level Switchboard restarts and never see frame timing. General audio language models score 14.2% (SALMONN) and 12.2% (LTU-AS) F1 on VCTK++ per SSDM. We do not rerun these and treat them as settled.
Proposed Experiments
The decisive check is the comparison between graph-marginal and fluent-CTC LoRA on clinical held-out speakers (nfvPPA and UCLASS). We measure repetition and prolongation recall, phonetic dysfluency F1, and boundary error in milliseconds. For boundary error, Neural LCS reports 17 ms on PPA (Seamless Dysfluent Speech Text Alignment), which we use as a target. The hypothesis fails if graph training does not beat fluent-CTC training on recall, or if it does so only by producing more false loops on fluent speech. We measure that with a false-positive rate on fluent read speech and on Buckeye conversational speech.
On simulated data we report WPER on simu-rep, simu-del and simu-ins and duration-aware phoneme error on VCTK++. The deletion set is the one to watch: deletions leave no acoustic trace, so we expect the smallest gains there, and we report it separately. On DisfluenTIMIT we report frame Overlap against the 0.67 figure.
A rank sweep crossed with the amount of clinical adaptation data (1, 5 and 20 speakers) tests the regularization claim. If low rank helps, its advantage over full tuning should be largest at the smallest data sizes and shrink as data grows. Emission calibration error on disfluent frames, compared before and after adaptation, tests whether the calibration-preservation effect seen in language models carries over to phoneme posteriors. All results use speaker-disjoint splits and three seeds, with confidence intervals bootstrapped over speakers, because clinical test sets of a few dozen patients make single-run differences of a few F1 points uninterpretable.
References
- Dysfluent WFST: A Framework for Zero-Shot Speech Dysfluency Transcription and Detection
- Enhancing General Face Forgery Detection via Vision Transformer with Low-Rank Adaptation
- On the Calibration of Large Language Models and Alignment
- Weakly-supervised forced alignment of disfluent speech using phoneme-level modeling
- Sparsely Shared LoRA on Whisper for Child Speech Recognition
- LoRA-Whisper: Parameter-Efficient and Extensible Multilingual ASR
- SSDM: Scalable Speech Dysfluency Modeling
- Unconstrained Dysfluency Modeling for Dysfluent Speech Transcription and Detection
- Towards Hierarchical Spoken Language Disfluency Modeling
- Boosting Disfluency Detection with Large Language Model as Disfluency Generator
- Seamless Dysfluent Speech Text Alignment for Disordered Speech Analysis