Severity-Gated Warping Adapters for Severe Dysarthric Speech Recognition

Speech recognition systems fail badly for people with dysarthria — the motor-speech impairment caused by Parkinson's disease, cerebral palsy, ALS, and stroke. Their speech stretches and compresses unpredictably, vowels collapse toward each other, and stop consonants leak into fricative-like sounds. Despite a wave of work adapting large pre-trained speech models to this population, the hardest group — speakers with "Very Low" intelligibility — is stuck: the best reported word error rates on the standard UASpeech benchmark for this bracket sit between 46% and 52% (Structured Speaker-Deficiency Adaptation of Foundation Models for Dysarthric and Elderly Speech Recognition, Enhancing Pre-trained ASR System Fine-tuning for Dysarthric Speech Recognition using Adversarial Data Augmentation, Exploring Self-supervised Pre-trained ASR Models For Dysarthric and Elderly Speech Recognition). These are the people who need assistive speech technology most, and for whom current systems are least usable.

This proposal imports a design principle from a field that at first glance has nothing to do with speech: pharmaceutical encapsulation for improving drug bioavailability. That field has converged on a three-part recipe for delivering fragile payloads through hostile environments — shield the payload, anchor it at the target tissue, and release its effect only when a disease-specific trigger is present. Layer-by-layer nanocoatings around probiotics protect the cells while leaving a thin, selectively permeable shell (Improved functionality of Ligilactobacillus salivarius Li01 in alleviating colonic inflammation by layer-by-layer microencapsulation), and pH-responsive hydrogels release their cargo only at the acidic pH of infected tissue while staying inert elsewhere (A Fungistatic Strategy Using a Shear-Thinning pH-Responsive CMCS-OHA-Lp/Lr Hydrogel for Vulvovaginal Candidiasis). No paper in the Lacuna index combines these two literatures; the connection here is a methodological transplant, not a physical one. The claim is that the protect–anchor–release recipe, translated into model architecture, addresses exactly the failure mode that keeps severe dysarthria recognition broken.

Core Claim

The residual errors on Very Low intelligibility speech come from local, non-linear distortions in time and frequency that global adaptation cannot repair. Saz et al. quantified this decades ago in the HMM era: pathological word durations vary by 25–38% around the mean (versus 15% for healthy speakers), vowel formant clusters collapse, and a model that warps time and frequency dynamically and locally — their MATE system — beat static global warping by 17.04% relative WER (Study of time and frequency variability in pathological speech and error reduction methods for automatic speech recognition). Yet modern foundation-model systems for dysarthria use only speaker-level, global adaptation: modular adapters that scale or shift features uniformly per speaker (Structured Speaker-Deficiency Adaptation of Foundation Models for Dysarthric and Elderly Speech Recognition), SVD-based speaker-basis features (Speaker Adaptation Using Spectro-Temporal Deep Features for Dysarthric and Elderly Speech Recognition), or GAN augmentation that imposes speaker-level spectral coloration (Enhancing Pre-trained ASR System Fine-tuning for Dysarthric Speech Recognition using Adversarial Data Augmentation). Nobody has put local dynamic time–frequency compensation inside a foundation model. The core claim: a thin, severity-gated module that applies a learned local time–frequency warp only on impaired segments, wrapped around a frozen speech foundation model, will cut Very Low intelligibility WER below the best reported 46.47% at single-model compute, while leaving fluent speech untouched.

The Encapsulation Analogy as Architecture

Each ingredient of the biomaterials recipe maps onto one architectural commitment. Shielding: the foundation backbone (HuBERT or Wav2vec2-Conformer) stays frozen, as Jin et al. showed it must under extreme data scarcity — full fine-tuning memorizes the quirks of the few training speakers, while modular LHUC and residual adapters that leave the backbone intact reduce WER by up to 3.01% absolute on UASpeech (Structured Speaker-Deficiency Adaptation of Foundation Models for Dysarthric and Elderly Speech Recognition). Our modules are thin shells on top of this frozen core, in the spirit of the ~10 nm probiotic nanocoating that protects without suffocating (Improved functionality of Ligilactobacillus salivarius Li01 in alleviating colonic inflammation by layer-by-layer microencapsulation). Anchoring: the warp offsets are conditioned on enrolled speaker representations, so compensation is anchored to each speaker's own motor pattern rather than a population average — the computational analogue of mucoadhesive anchoring that stops payloads from washing away under shear, and kin to Yee et al.'s per-speaker latent templates (Latent Phrase Matching for Dysarthric Speech). Triggered release: the compensation activates only where the speech is actually impaired, mirroring hydrogels that release 83.50% of their probiotic load at infection pH 5.5 while remaining inert at systemic pH 7.4 (A Fungistatic Strategy Using a Shear-Thinning pH-Responsive CMCS-OHA-Lp/Lr Hydrogel for Vulvovaginal Candidiasis) — selectivity is the whole point of the trigger.

Concretely, a light gate network predicts a per-frame impairment probability $g_t \in [0,1]$ from frozen foundation features, and a warp module transforms the acoustic input before it enters the backbone:

$$\tilde{x}(t,f) = x\big(t + g_t\,\delta_t,\; f\,(1 + g_t\,\delta_f)\big)$$

where $\delta_t$ and $\delta_f$ are speaker-conditioned, learned time and frequency warp offsets. When $g_t = 0$ — a fluent segment — the input passes through unchanged, giving selective permeability by construction. This is MATE's discrete state-expanded warping (Study of time and frequency variability in pathological speech and error reduction methods for automatic speech recognition) re-cast as a continuous, differentiable flow field sitting outside a frozen neural encoder, so it can be trained end-to-end with the adapter stack by backpropagation through the sampling grid.

Research Program

Three things get built and analyzed. First, the gate: a small network trained jointly with a severity-classification auxiliary head, following the multi-task severity signal that Wang et al. found useful for regularizing fine-tuning (Enhancing Pre-trained ASR System Fine-tuning for Dysarthric Speech Recognition using Adversarial Data Augmentation); an interpretability analysis will correlate gate activation with the physically documented distortion phenomena — temporal jitter, vowel formant collapse, and the consonant spirantization that Moro-Velázquez et al. measured directly (the vowel /u/ substitution rate is 12 times higher in Parkinson's speakers than controls; Study of the Performance of Automatic Speech Recognition Systems in Speakers with Parkinson's Disease). If the gate fires on exactly the segments where these distortions live, the mechanism is doing what the design claims. Second, the warp: speaker-conditioned offset fields with a small vocabulary of warp bases, so a speaker's anchor needs only a few enrolled utterances — relevant because the CUHK system reports useful adaptation from as little as 3.06 seconds of speech (Recent Progress in the CUHK Dysarthric Speech Recognition System). Third, the integration: warp plus LHUC/residual adapters on a frozen backbone, with the same test-time unsupervised adaptation protocol as Jin et al., so the comparison against their system isolates the warp as the only new ingredient.

Baselines

The "do nothing" reference point is an off-the-shelf recognizer applied unadapted to dysarthric speech. On the Spanish Neurovoz corpus, an end-to-end CTC/attention system scores 47.0% WER on Parkinson's speech versus 36.7% on healthy controls (Study of the Performance of Automatic Speech Recognition Systems in Speakers with Parkinson's Disease), and on the Saz corpus healthy-trained systems exceed 20–25% WER on pathological speech versus roughly 7% on healthy speech (Study of time and frequency variability in pathological speech and error reduction methods for automatic speech recognition). This establishes the size of the mismatch we are attacking and the floor any adaptation must clear.

The strongest single-model competitor is the structured speaker-deficiency adaptation of Jin et al., which inserts LHUC and residual adapter blocks into frozen HuBERT and Wav2vec2-Conformer models and reports 19.45% overall WER on UASpeech, 49.34% on the Very Low intelligibility bracket, and 33.17% on unseen words (Structured Speaker-Deficiency Adaptation of Foundation Models for Dysarthric and Elderly Speech Recognition). It is the strongest because it is architecturally the closest to ours — same frozen backbones, same adapter philosophy — so beating it isolates the local warp rather than any incidental advantage. We will re-run it ourselves on identical data splits, checkpoints, and tuning budget (same adapter parameter count for our gate-plus-warp modules, same hyperparameter search protocol), and beating it means a statistically significant WER reduction on the Very Low bracket with no regression on the High intelligibility bracket or on DementiaBank Pitt. The strongest reported number of any kind on UASpeech is 16.53% overall and 46.47% on Very Low, from Wang et al.'s cross-pass multi-model rescoring pipeline (Enhancing Pre-trained ASR System Fine-tuning for Dysarthric Speech Recognition using Adversarial Data Augmentation); that system uses an ensemble and second-pass rescoring, so it cannot be matched at equal compute. We treat its 46.47% Very Low figure as the target number to beat at single-forward-pass compute, and we will measure real-time factor to make the compute gap explicit rather than pretend it away — the edge-deployment constraint flagged by Hu et al. is real (Exploring Self-supervised Pre-trained ASR Models For Dysarthric and Elderly Speech Recognition).

The key ablation replaces the severity gate with an always-on constant, giving the same warp module applied uniformly to every frame — that is, global speaker-level warping in the spirit of standard Vocal Tract Length Normalization. Same data, same budget, same backbone. If always-on warping matches gated warping, the entire "triggered release" story collapses and the proposal's central claim is wrong; this ablation is the honest test of the ingredient that makes the work more than "add warping to adapters."

Why not just…?

Why not just apply global VTLN or time-stretching? Because the distortions are local and non-linear, and the evidence is already in the file. Saz et al. measured temporal variability of 25–38% around the mean in pathological speech and showed that dynamic, state-level warping (MATE) reduced WER by 17.04% relative where static global warping could not (Study of time and frequency variability in pathological speech and error reduction methods for automatic speech recognition); and Liu et al.'s tempo and vocal-tract-length perturbations, while helpful, were consistently complemented rather than replaced by stronger methods (Recent Progress in the CUHK Dysarthric Speech Recognition System). Our always-on ablation is precisely the experiment that settles this in the foundation-model setting — if it wins, we report that the gating was unnecessary.

Why not just fine-tune the whole foundation model end-to-end? Jin et al. provide the direct negative result: full fine-tuning on small dysarthric corpora overfits the idiosyncrasies of the few training speakers, while keeping the backbone frozen with modular adapters reduced WER by up to 3.01% absolute (10.86% relative) on UASpeech and 1.50% absolute on DementiaBank Pitt (Structured Speaker-Deficiency Adaptation of Foundation Models for Dysarthric and Elderly Speech Recognition). Data scarcity is not going away — recording is physically exhausting for these patients — so the frozen-backbone constraint is the right one. We adopt their protocol rather than argue with it.

Why not just use personalized template matching, as in Latent Phrase Matching? For a severe user with a fixed command set, LPM is a genuine competitor — it maintains roughly 70% accuracy for severe dysarthria where commercial ASR fails, and a 60% relative recall improvement on an internal 32-speaker dataset (Latent Phrase Matching for Dysarthric Speech). But Yee et al. themselves show accuracy degrading as vocabulary approaches 50 phrases due to embedding collisions, and the method cannot produce open-vocabulary output at all. Our experiments include LPM as a baseline on the severe bracket at small and large vocabulary sizes; if LPM wins below 50 phrases and our system only wins above, that is a useful honest boundary for deployment, and the unseen-words split on UASpeech (where Jin et al. report 33.17% WER) tests whether phonetic generalization survives.

Why not just generate more synthetic dysarthric data with GANs? Wang et al.'s GAN augmentation gave a 2.01% absolute WER reduction for Wav2vec2.0 fine-tuning (Enhancing Pre-trained ASR System Fine-tuning for Dysarthric Speech Recognition using Adversarial Data Augmentation), so this is a real threat and we will run it as an additional baseline. But the augmentation transfers speaker-level spectral signatures — it cannot manufacture the segment-level temporal jitter of 25–38% that motivates local compensation, and the literature itself cautions that synthetic dysarthric speech is a supplement, not a substitute for authentic data (Synthetic Dysarthric Speech: A Supplement, Not a Substitute for Authentic Data in Dysarthric Speech Recognition). The two approaches are also complementary in principle; if combining them helps, we will say so rather than defend a turf boundary.

Decisive Checks

The proposal succeeds if, on UASpeech with matched data and compute, the gated-warp adapter system reduces Very Low intelligibility WER below the 46.47% reported by Wang et al. (Enhancing Pre-trained ASR System Fine-tuning for Dysarthric Speech Recognition using Adversarial Data Augmentation) and below our re-run of Jin et al.'s system (Structured Speaker-Deficiency Adaptation of Foundation Models for Dysarthric and Elderly Speech Recognition), with no regression on the overall test set (baseline 19.45%) or on unseen words (33.17%), and if the always-on ablation shows a clear gap behind the gated version. A second decisive check is mechanism-level: gate activation should correlate with the documented physical distortions — the temporal jitter and formant collapse quantified by Saz et al. and the spirantization patterns documented by Moro-Velázquez et al. — rather than firing uniformly, because a gate that is always high is just a bias term wearing a costume. Generalization is checked on DementiaBank Pitt conversational speech (best reported 17.45% WER from Jin et al.; Structured Speaker-Deficiency Adaptation of Foundation Models for Dysarthric and Elderly Speech Recognition) and on the Saz Spanish corpus, where MATE's 17.04% relative reduction gives a direct, language-crossing comparison against the original dynamic-warp idea our module descends from (Study of time and frequency variability in pathological speech and error reduction methods for automatic speech recognition). No reported result found on Lacuna for any prior system combining local dynamic warping with foundation-model adapters, which is exactly the hole this work fills.