People with dysarthria, slurred or distorted speech caused by conditions such as cerebral palsy, ALS or Parkinson's disease, are badly served by speech recognizers. One of the more effective recent fixes is surprisingly simple. The user records a few examples of each word. The system averages their acoustic embeddings into a "prototype" for that word, then labels new speech by finding the nearest prototype. The cost is that every word must be recorded by the user first. That rules out the case that matters most in practice: a user who needs a medication name, a colleague's surname or a technical term they have never recorded, and who may tire quickly when asked to record at all.
We propose to build prototypes for unrecorded words out of the user's own recorded sounds. From a handful of enrolled words, the system collects how this particular speaker actually produces each phoneme, meaning each basic speech sound such as /p/ or /æ/. When a new word arrives as text, it is converted to a phoneme string. The speaker's phoneme snippets are strung together in that order, and the sequence is passed through the same network that embeds real audio. The result is a word prototype in the speaker's own acoustic voice, and it can be compared directly with what the speaker says. Because the audio branch and the composed-text branch share one set of weights, this is a Siamese network: twin branches with identical parameters, trained so that matching pairs land close together. The claim is that this composition lets a dysarthric speaker's domain vocabulary grow with no further recording, and that it beats both text-to-speech synthesis of the new words and phonetic spelling correction applied after recognition.
Core Claim
Zero-shot recognition of out-of-vocabulary words for a dysarthric speaker is feasible if the speaker's systematic distortions are carried into the word prototype at the level of phonemes, rather than assumed away by a canonical pronunciation. Three components make this concrete. The first is a per-speaker phone table, built by aligning enrolled audio to phonemes. The second is a composition step that turns a text pronunciation into a pseudo-acoustic sequence using that table and the speaker's average phone durations. The third is a weight-shared sequence encoder that embeds both real and composed sequences, trained so that a composed word lands near real utterances of that word.
What is new is the source of the prototype. It is neither canonical text, as in healthy-speech keyword spotting, nor the user's recording of the target word, as in existing dysarthric prototype methods. It is the user's recordings of other words, recombined.
Where This Sits
Two lines of work motivate the proposal by showing that the pieces work separately. Enhancing Dysarthric Speech Recognition for Unseen Speakers via Prototype-Based Adaptation (PB-DSR) fine-tunes HuBERT with a recognition loss plus a contrastive loss. It then averages per-word embeddings from a speaker's support set and reports an average WER reduction of 15.59% over speaker-independent models for unseen speakers. It needs audio for every target word, however, and it builds prototypes from only the first HuBERT frame. We remove the first requirement and replace the second with a full sequence encoder. PhonMatchNet matches audio against text phoneme sequences for user-defined keywords and reaches 96.8% zero-shot accuracy on Google Speech Commands. It does this only on healthy speech, using canonical grapheme-to-phoneme output, and that is exactly the assumption dysarthria breaks. We keep its cross-modal matching idea but ground the text side in the speaker's realized phones.
Other work supplies the method. DyPCL shows that dysarthric errors are largely phonemic. Its phoneme-level contrastive learning with dynamic alignment from Connectionist Temporal Classification (CTC, a training loss that aligns frames to output symbols without frame labels) lowers WER for "Very Low" intelligibility speakers from 58.49% to 49.45%. We reuse CTC alignment to cut enrolled words into phone snippets, and DyPCL-style phonetic hard negatives to train the encoder. Prototypical Networks supplies the squared-Euclidean nearest-mean rule and episodic training. Meta-Baseline supports pre-training the encoder normally before episodic fine-tuning.
Some work constrains the design. Latent Phrase Matching for Dysarthric Speech matches streaming embeddings against enrolled templates with dynamic time warping. It works for severe speakers, reaching about 70% accuracy, but it degrades as the vocabulary grows toward 50 phrases. Any proposal therefore has to be evaluated as vocabulary size increases, not at one small size. Dysarthric Speech Recognition Based on Deep Metric Learning gets large gains from angular-margin metric learning, but it sums its features over time, which discards the order of sounds. Our composed sequences rely on that order, so pooling has to happen after a temporal encoder. Synthetic Dysarthric Speech: A Supplement, Not a Substitute finds synthetic dysarthric speech "too clean": substituting it for real data raises CER on CDSD from 35.1% to 37.9%. That makes synthesis a natural but suspect way to get prototypes for new words, and we test it directly.
On the vocabulary-gap side, RSpell fixes domain terms after the fact by fuzzy pinyin retrieval from a lexicon. The SAP Challenge reports that LLM post-correction tends to smooth rare words into common ones. Biomedical Language Models are Robust to Sub-optimal Tokenization shows that reworking the tokenizer barely moves downstream scores. Together these suggest that the bottleneck for rare words in disordered speech lies in the acoustic evidence, not the text vocabulary. That is the premise we test.
Research Program
The encoder is a fine-tuned HuBERT front end followed by a small temporal transformer, pooled into one vector, and the same weights embed both branches. Real audio enters as frame features. A composed word enters as a sequence of phone snippets, each a stored mean feature vector for that phone from this speaker, repeated for the speaker's mean duration of that phone. This works because both branches start from the same frame-level feature space. Training runs in episodes on training speakers, and each episode simulates the deployment split. Some of the speaker's words act as "enrolled" and are used only to build the phone table. Held-out words are represented only by composed prototypes and must be recognized from real queries. The loss is prototype cross-entropy over squared Euclidean distance, $-\log \frac{\exp(-\|f(q) - f(\tilde{c}_w)\|^2)}{\sum_{w'} \exp(-\|f(q) - f(\tilde{c}_{w'})\|^2)}$. Here $f$ is the shared encoder, $q$ is a real query utterance, and $\tilde{c}_w$ is the composed sequence for word $w$. Negatives are drawn with phonetic-distance curricula as in DyPCL.
Phones that the speaker has never enrolled fall back to a speaker-independent dysarthric phone mean, shifted by the speaker's average offset. We will report how often this fallback fires. When a word does have recordings, its composed prototype and its real-audio prototype can be averaged, which tells us how much enrollment is still worth once composition exists.
Baselines
The naive reference point is a speaker-independent HuBERT model fine-tuned with CTC, trained on the same training speakers as every other system. Its output is constrained to the test vocabulary by restricting decoding to that lexicon. Any proposal must beat this to justify itself.
The strongest existing method that solves the same zero-shot problem is a two-stage pipeline. A CTC phone recognizer is fine-tuned on dysarthric training speakers. Its phone string for the query is then matched against the domain lexicon by weighted edit distance, with substitution costs taken from that speaker's phone confusion matrix estimated on the enrolled words. This is RSpell's phonetic retrieval idea, moved from pinyin to acoustics and personalized. It is the most serious competitor because it uses exactly the same speaker information and needs no new architecture.
The second strong competitor gives PB-DSR prototypes for new words through synthesis. A TTS system fine-tuned on dysarthric speech generates the unseen words, and voice conversion maps them toward the target speaker. PB-DSR's nearest-prototype rule is then applied on top. A third competitor is PhonMatchNet-style matching against canonical text, retrained on the same dysarthric training data so the comparison isolates canonical versus speaker-realized phones. A fourth is a fine-tuned Whisper model with the domain lexicon supplied as a decoding prompt.
PB-DSR with real recordings of the target words serves as an oracle upper bound, not as a competitor. No paper in the file reports zero-shot unseen-word accuracy on dysarthric speech, so there is no published number to beat on this task; we found no reported result on Lacuna for it. For context, the CUHK system reports 69.4% word accuracy on UASpeech for the closed vocabulary (Development of the CUHK Dysarthric Speech Recognition System), and PB-DSR reports its 15.59% average WER reduction with audio support.
Fairness rules apply to all systems. Every system uses the same HuBERT checkpoint where applicable, the same training speakers, the same enrolled words per test speaker, the same leading and trailing silence stripping (worth 6.9% absolute accuracy in the CUHK system), and the same hyperparameter search budget on development speakers. Beating the field means two things on held-out speakers. The method must have higher unseen-word accuracy than the confusion-weighted retrieval pipeline in the Low and Very Low intelligibility groups, with confidence intervals from a speaker-level bootstrap that exclude zero. And it must close at least half of the gap between the speaker-independent reference and the audio-enrolled PB-DSR oracle.
Three ablations isolate the key ingredient. The first replaces the speaker's phone table with one from healthy control speakers, which tests whether the speaker grounding matters. The second replaces the sequence encoder with first-frame or mean pooling, which tests whether temporal order matters. The third unties the weights between the two branches, which tests whether weight sharing helps.
Why not just…?
Why not just fine-tune Whisper or a Conformer on dysarthric data and supply the domain words as a prompt? Fine-tuning helps mild speakers a great deal: in the SAP Challenge, WER fell to 8.11% from the 17.82% of zero-shot Whisper. Severe speakers stay far worse, as DyPCL's 58.49% CTC WER for the Very Low group shows. A prompt adds a word to the text side, but it gives no acoustic model of how this speaker distorts that word. We run this baseline and break results down by intelligibility group. We expect it to stay competitive for High-intelligibility speakers and to fall short for Low and Very Low ones. If it does not, the proposal's case shrinks to severe speakers only, and we will say so.
Why not just recognize phones and fuzzy-match them against the lexicon? This is the real threat. A phone recognizer combined with a speaker-specific confusion matrix captures much of the same information as our phone table. There are two reasons to expect it to lose. Dysarthric distortions involve timing and blending across neighboring sounds, which an edit distance over discrete phones cannot represent. And errors in the first-stage phone string are committed before the lexicon is consulted. Neither reason is proven. The head-to-head comparison across vocabulary sizes, with matched enrollment, settles it. If the pipeline wins, the honest conclusion is that discrete phonetic retrieval is the right tool and the embedding machinery is unnecessary.
Why not just synthesize the new words in the speaker's voice and use ordinary prototypes? The CDSD results suggest that synthetic dysarthric speech lacks the pitch and loudness instability of real speech. A prototype averaged from such samples should therefore sit too close to typical pronunciations. Our composed prototypes are built from real snippets of the speaker, so they inherit that instability. The synthesis baseline tests this directly. A secondary analysis measures how far synthetic and composed prototypes each lie from the oracle audio prototype.
Why not just use template matching with dynamic time warping over the composed sequence, with no training? Latent Phrase Matching already shows that template matching degrades as vocabulary grows. We include a no-training variant that warps queries against composed sequences, which tells us how much of the gain comes from training the encoder episodically.
Proposed Experiments
The primary benchmark is UASpeech. All of its speakers record the same word list, which splits into digits, the radio alphabet, computer commands, common words and uncommon words, so it supports a clean zero-shot protocol. Test speakers are held out entirely in a leave-speakers-out design. For each test speaker, only digits, alphabet, commands and common words are available for enrollment. The uncommon words stand in for domain terms. They are removed from encoder training for all speakers, so neither the model nor the speaker has heard them. We report word accuracy by intelligibility group as the candidate vocabulary grows from 50 to the full uncommon list. This directly probes the scaling weakness reported for template matching.
Two decisive checks follow. The first sweeps enrollment budget from 20 to 150 words, to find how much recording the method needs before it overtakes the confusion-weighted pipeline. The second measures accuracy on uncommon words whose phones fall back to the speaker-independent table, which shows whether coverage gaps explain the failures.
For a less proxy-like domain test, we assemble a medical-term subset from SAP-240430 prompts, train on the remaining SAP speakers, and evaluate term-level recall within utterances by spotting composed prototypes against sliding windows. Because SAP is about 75% Parkinson's data, results are reported per etiology. Finally, EasyCall, which is Italian, tests whether phone composition transfers across languages, using its command and non-command split. The non-commands double as an open-set rejection test, reported as AUROC. Here distance thresholds on composed prototypes are compared with the dummy prototypes of Dummy Prototypical Networks, which reached 86.7% AUROC on healthy splitGSC.
References
- Enhancing Dysarthric Speech Recognition for Unseen Speakers via Prototype-Based Adaptation
- PhonMatchNet: Phoneme-Guided Zero-Shot Keyword Spotting for User-Defined Keywords
- DyPCL: Dynamic Phoneme-level Contrastive Learning for Dysarthric Speech Recognition
- Prototypical Networks for Few-shot Learning
- Meta-Baseline: Exploring Simple Meta-Learning for Few-Shot Learning
- Latent Phrase Matching for Dysarthric Speech
- Dysarthric Speech Recognition Based on Deep Metric Learning
- Synthetic Dysarthric Speech: A Supplement, Not a Substitute for Authentic Data in Dysarthric Speech Recognition
- RSpell: Retrieval-Augmented Framework for Domain Adaptive Chinese Spelling Check
- The Interspeech 2025 Speech Accessibility Project Challenge
- Biomedical Language Models are Robust to Sub-optimal Tokenization
- Development of the CUHK Dysarthric Speech Recognition System for the UA Speech Corpus
- Dummy Prototypical Networks for Few-Shot Open-Set Keyword Spotting