Speech recognizers for people with dysarthria (slurred or effortful speech caused by neuromotor damage) are usually personalized by adding a small set of trainable parameters, called an adapter, to a large pretrained model and fitting it on a few minutes of the user's speech. Most work picks one adapter size and placement for everyone. That choice is made by averaging error rates over a test cohort, and the system then ships to individual patients. This note proposes to treat each speaker as their own small experiment. For every speaker we would measure how recognition error changes as adaptation capacity grows. We would then test whether a cheap check run only on that speaker's enrollment data can pick a good capacity, without a cohort of similar patients to learn from. That includes picking zero adaptation when adaptation would hurt.
Two observations motivate the work. In healthy telephone speech, Leveraging speaker-dependent variation of adaptation found that the best number of adaptation classes differs sharply across speakers. One cluster of speakers did best with no adaptation at all. A per-speaker oracle would have cut word error rate (WER, the fraction of words misrecognized) from 18.6% to 17.4%. Their cohort-trained predictor recovered only 0.4 points of that gain. In clinical and assistive work, single-participant studies such as Restoring brain-to-text communication in a person with dysarthria from pontine stroke using an intracortical brain-computer interface show that something works for one person. They cannot show that it holds as a pattern across people. Disordered-speech adaptation papers sit between these two problems. They report cohort averages, so individual over-adaptation is hidden, while their deployment target is a single person whose profile may match no cohort bucket.
Core Claim
We claim three things, each testable. First, on dysarthric corpora the best adaptation capacity varies across speakers more than intelligibility class explains, and a sizable minority of speakers do best with little or no adaptation. Second, a within-speaker selection rule can recover a meaningful share of the gap between a fixed cohort-tuned capacity and the per-speaker oracle. The rule must validate on held-out words the speaker never produced during adaptation, not on held-out repetitions of the same words. Third, part of what high-capacity adapters learn on these corpora is the recording channel rather than articulation. A channel-only control can measure that share.
The word-disjoint validation is the key ingredient, and the reason is specific. Over-adaptation in recognition shows up as memorizing the speaker's renditions of enrollment words. Random utterance splits reward that memorization, because the same word types appear on both sides of the split. The cohort numbers already hint at this. Structured Speaker-Deficiency Adaptation of Foundation Models for Dysarthric and Elderly Speech Recognition reports 19.45% overall WER on UASpeech but 33.17% on unseen words.
Relation to Prior Work
Three papers supply the method and motivate the gap. Leveraging speaker-dependent variation of adaptation selects regression-tree size per speaker with a classifier trained on 464 speakers of healthy speech. We remove the cohort dependence and move to adapters on foundation models for atypical speech, where no comparable training cohort exists. Structured Speaker-Deficiency Adaptation of Foundation Models for Dysarthric and Elderly Speech Recognition separates speaker-identity adapters from deficiency-level adapters in HuBERT and Wav2vec2-conformer backbones. It shows that early placement works best, but it assigns deficiency adapters by four categorical intelligibility levels and evaluates averages over 16 speakers. We reuse its adapter types (LHUC, which rescales hidden units, and residual adapter blocks) and its placement axis. We ask whether its single cohort-tuned configuration is right for each speaker. Recent Progress in the CUHK Dysarthric Speech Recognition System shows that Bayesian adaptation works with about 3 seconds of speech. It supplies the main competing hypothesis: a well-set prior might make explicit capacity selection unnecessary.
Two papers constrain the design. Regularized-MLLR speaker adaptation for computer-assisted language learning system shows that unconstrained high-capacity adaptation can learn the very errors one wants to detect. Its fix is to restrict the transform to a subspace spanned by native teachers. That is the right concern for pronunciation scoring, which is why we restrict the claim to recognition, where the target is the intended word. On using the UA-Speech and TORGO databases to validate automatic dysarthric speech classification approaches finds an 11.3 dB SNR gap between control and dysarthric recordings. Models in that study classify speakers at 96% to 100% accuracy from silence alone. Any per-speaker adapter gain on these corpora is therefore partly suspect, and we build a control for it.
Finally, Personalization of Large Foundation Models for Health Interventions frames the tension between population priors and N-of-1 validation, but only conceptually, with a simulated case. We turn that framing into a concrete protocol. The rule is designed on one corpus and frozen. It is then applied to others, so that a per-speaker result becomes a claim about a pattern rather than a single existence proof. Speaker Identity Preservation in Dysarthric Speech Reconstruction by Adversarial Speaker Adaptation is relevant as a cautionary example. It uses one fixed adapter capacity and evaluates on four speakers, which is exactly the kind of result this protocol is meant to stress-test. Because it addresses reconstruction rather than recognition, we do not use it as a baseline.
Research Program
We build a capacity ladder on a single pretrained backbone, following the structured-adapter setup above. The rungs are ordered, and each adds trainable parameters. Rung zero is no adaptation. The next rung is a single global LHUC scaling vector after the convolutional feature encoder. Higher rungs add LHUC at more Transformer layers, then residual adapter blocks of increasing width, and the top rungs add longer adaptation schedules. The order follows the placement finding that early layers matter most for impairment. Every speaker is adapted at every rung, giving a per-speaker capacity curve on held-out test words.
The selection rule uses only the speaker's enrollment data. That data is split by word type into a fit portion and a validation portion. The adapter is fit at each rung on the fit portion. The rule picks the rung with the lowest validation WER, subject to a one-standard-error rule from a bootstrap over validation words, which prefers the smaller rung when differences are within noise. The selected rung is then refit on all enrollment data. The rule's hyperparameters are fixed on UASpeech training speakers and then frozen.
For the channel control, we fit an adapter of matched capacity whose only input statistics come from non-speech segments. Concretely, this is a global feature-normalization transform estimated from silence. We report how much of each rung's gain this channel-only adapter reproduces.
Analysis treats speakers as the unit. For each speaker we report the oracle rung, the selected rung, and bootstrap intervals over test words. A hierarchical regression of oracle rung on intelligibility class and enrollment amount tests whether severity bucket explains the variation, which is the implicit assumption behind categorical deficiency adapters.
Baselines
The naive reference point is the unadapted backbone fine-tuned on the training speakers. This is rung zero for everyone. It matters because it defines which speakers adaptation actually hurts.
The strongest existing method is the structured speaker-deficiency configuration from Structured Speaker-Deficiency Adaptation of Foundation Models for Dysarthric and Elderly Speech Recognition. It reports 19.45% overall WER on the UASpeech 16-speaker test set, 49.34% on the very-low-intelligibility group, 33.17% on unseen words, and 17.45% on DementiaBank Pitt. It is the strongest competitor because it is already tuned for capacity and placement at the cohort level. We will reimplement it on the same backbone and splits and apply it at its fixed configuration to every speaker. Because a reimplementation may not hit 19.45%, we compare against our own reproduction, and we report the published figure alongside it.
The second baseline is a cohort-trained capacity predictor in the style of Leveraging speaker-dependent variation of adaptation. It is a bagged decision-tree classifier that predicts each speaker's oracle rung from acoustic likelihood and decoding statistics, trained leave-one-speaker-out on the other dysarthric speakers. On healthy telephone speech it recovered one third of the oracle gap (18.6% to 18.2% against an oracle of 17.4%). Recovering a comparable or larger share without any cohort is our bar.
The third baseline is Bayesian LHUC at the top rung, following Recent Progress in the CUHK Dysarthric Speech Recognition System. That paper reports 25.21% WER on UASpeech.
The key ablation replaces the word-disjoint validation split with a random utterance split. Everything else is held identical. All methods get the same enrollment data, backbone, decoding setup, and the same number of adaptation runs per speaker. The cohort predictor and Bayesian prior get their hyperparameters tuned on the same training speakers our rule uses.
Beating the field means two things. Our rule must give lower mean per-speaker WER than the fixed structured configuration, with significance under a paired test over speakers. It must also reduce the number of speakers made worse by adaptation.
Why not just…?
Why not just use the cohort-tuned fixed adapter, since it already holds the best published number? A cohort average can hide speakers who are being hurt. The healthy-speech evidence shows a whole cluster for whom any adaptation added errors. The per-speaker curves will show directly whether such speakers exist on UASpeech and TORGO. If every speaker's oracle rung matches the fixed configuration, the proposal's premise fails, and we would report that.
Why not just train a granularity predictor across dysarthric speakers, as was done for telephone speech? Dysarthric corpora have tens of speakers rather than hundreds. Severity, etiology, and recording setup are also entangled. A predictor trained on such a cohort may simply learn the channel, which the silence-classification result makes plausible. We run this predictor as a baseline. If it matches our rule, the conclusion is that small cohorts suffice, which is useful to know. But it would still need a cohort, which a new patient with an unusual profile, such as the pontine-stroke participant, does not come with.
Why not just use Bayesian adaptation at full capacity and let the prior do the shrinking? This is the real threat. A good prior could make selection redundant. The decisive comparison is Bayesian LHUC at the top rung against our selected rung, broken out by intelligibility group and by speakers with the least enrollment data. If Bayesian adaptation matches selection everywhere, we will say so. The remaining contribution would then be the channel-share measurement and the per-speaker reporting protocol. A hybrid in which our rule selects among Bayesian rungs is a natural follow-up.
Why not just collect more enrollment speech? More speech does not remove the memorization pathway if it repeats the same word list, and recording is tiring for these speakers. We measure this directly by subsampling enrollment data per speaker and tracking how the oracle rung and the selection error change.
Proposed Experiments
The primary experiment runs the full ladder on the UASpeech test speakers with the standard block-based split, so that unseen-word performance is measured. Metrics are mean per-speaker WER, WER on unseen words, the fraction of speakers worse than rung zero, and the oracle-gap recovered:
$$R = \frac{W_{\text{fixed}} - W_{\text{selected}}}{W_{\text{fixed}} - W_{\text{oracle}}},$$
where $W$ denotes mean per-speaker WER under the fixed structured configuration, our selected rung, and the per-speaker oracle rung, respectively. On healthy speech the cohort predictor reached $R$ of about one third.
The replication experiment applies the frozen rule, unchanged, to TORGO and to DementiaBank Pitt. This turns a UASpeech result into a cross-corpus pattern. For reference, Personalized Adversarial Data Augmentation for Dysarthric and Elderly Speech Recognition reports up to 0.91 points absolute WER reduction on TORGO and 3.0 points on DementiaBank. No per-speaker capacity result was found on Lacuna for either corpus.
The decisive checks are four.
The word-disjoint rule should achieve clearly positive $R$, while the random-split ablation picks larger rungs and achieves lower or negative $R$. If the two are indistinguishable, the memorization explanation is wrong.
The oracle rung should vary within intelligibility classes about as much as across them. If severity class alone predicts the oracle rung, categorical deficiency adapters are vindicated and per-speaker selection adds little.
The channel-only adapter should reproduce a measurable fraction of high-rung gains on UASpeech, and that fraction should be smaller on corpora with more uniform recording. A near-zero fraction would remove the artifact concern.
The result must hold on TORGO and DementiaBank with no retuning. Success on UASpeech alone is still a single-corpus finding, and we would frame it that way.
References
- Leveraging speaker-dependent variation of adaptation
- Restoring brain-to-text communication in a person with dysarthria from pontine stroke using an intracortical brain-computer interface
- Structured Speaker-Deficiency Adaptation of Foundation Models for Dysarthric and Elderly Speech Recognition
- Recent Progress in the CUHK Dysarthric Speech Recognition System
- Regularized-MLLR speaker adaptation for computer-assisted language learning system
- On Using the UA-Speech and Torgo Databases to Validate Automatic Dysarthric Speech Classification Approaches
- Personalization of Large Foundation Models for Health Interventions
- Speaker Identity Preservation in Dysarthric Speech Reconstruction by Adversarial Speaker Adaptation
- Personalized Adversarial Data Augmentation for Dysarthric and Elderly Speech Recognition