Personalized Phonetic Confusion Costs from a Handful of Words

People with severe motor speech disorders (dysarthria) mispronounce sounds, but they tend to do it in their own consistent way. One speaker may devoice every "b" into "p", and another may drop final consonants. Speech recognizers trained on typical speech treat these errors as noise and fail. The usual fixes pull in two directions. One retrains the acoustic model on the new speaker, which needs more recordings than a fatigued patient can give. The other stores whole-phrase audio templates, which work for a few enrolled commands but cannot recognize a command the user never recorded. This proposal takes a third route. Keep a frozen, general-purpose phone recognizer. Adapt only a small, discrete table saying how likely this particular speaker is to turn each intended sound into each recognized sound. Then spot any text command by fuzzy matching against the recognizer's competing hypotheses, using that table as the matching cost.

The central difficulty is that a per-speaker table of sound-to-sound confusions has on the order of a thousand entries, and a handful of enrollment words fills only a few of them. We propose to estimate the table by shrinkage. Each speaker's sparse counts are pulled toward a prior built from two sources: confusions pooled across other dysarthric speakers, and articulatory similarity between sounds, meaning sounds made at the same place in the mouth or in the same manner are more confusable. How strongly the table trusts the speaker's own counts is itself estimated from the data. For a mild speaker the table should stay close to "no confusion," which directly targets the known failure of earlier confusion models on high-intelligibility speakers.

Core Claim

With 5 to 20 enrollment words, personalizing the fuzzy-matching costs in a discrete phone-confusion space, under an articulatory and population prior, will detect commands more accurately than acoustic adaptation of the recognizer. It will also detect commands the user never recorded, which whole-phrase templates cannot do. Because matching operates on phone symbols from a frozen encoder rather than on raw audio, it should be much less able to exploit the recording-condition shortcuts that plague small clinical speech corpora.

This builds most directly on Modelling confusion matrices to improve speech recognition accuracy, with an application to dysarthric speech. That paper built per-phoneme "metamodels" mapping recognized to intended phones for each dysarthric speaker. It beat standard acoustic adaptation (MLLR) with as few as 4 sentences (about 50% vs 39.9% accuracy for one speaker), but it degraded relative to acoustic modelling for mild speakers. It was tested only on the small NEMOURS nonsense-sentence corpus with a decade-old phone recognizer and 1-best phone strings. That result motivates the proposal: discrete confusion adaptation is more sample-efficient than acoustic adaptation. Our design differs in three ways. The metamodel is estimated per speaker with no cross-speaker or articulatory prior, so we add shrinkage whose strength adapts to severity. It decodes over a sentence grammar, whereas we do open-vocabulary spotting. And it uses a speaker-independent phone recognizer trained on read typical speech, whereas we use a frozen self-supervised encoder.

Research Program

Phone recognizer. The front end is a phone recognizer built as a small classification head over frozen self-supervised speech representations, trained with CTC (a loss that lets a model emit sharp, sparse phone posteriors without frame alignments) on typical speech only. Two prior results motivate freezing. Unrestricted Vocabulary Keyword Spotting Using LSTM-CTC showed that CTC posteriors yield compact phone lattices supporting open-vocabulary spotting by minimum edit distance, at 3.6% equal error rate on WSJ0 with 813K parameters. We reuse that search design. On using the UA-Speech and TORGO databases to validate automatic dysarthric speech classification approaches found that classifiers trained on these corpora reach 96–100% accuracy from silence alone, because the corpora carry an 11.3 dB signal-to-noise gap between groups. Frozen wav2vec 2.0 features were the notable exception. This finding constrains the design: nothing trained on the small target corpus should see raw audio.

Enrollment counts. During enrollment the user speaks $k$ words whose text is known. A grapheme-to-phoneme converter gives the intended phones. An EM-style soft alignment between intended phones and the recognizer's phone lattice produces expected counts $n_s(a,b)$: how often speaker $s$'s intended phone $a$ surfaced as recognized phone $b$, with $b$ allowed to be "deleted" and insertions counted separately.

Shrinkage estimate. The personalized confusion probability is

$$ P_s(b \mid a) = \frac{n_s(a,b) + \alpha_s\,\pi(b \mid a)}{n_s(a) + \alpha_s}, $$

where $n_s(a)$ is the total count for intended phone $a$ and $\alpha_s$ is a per-speaker trust parameter chosen by maximizing the marginal likelihood of held-out enrollment words. The prior $\pi(b \mid a)$ is a learned mixture of three parts: a near-identity matrix, pooled confusions from training speakers, and a kernel over articulatory features (place, manner, voicing), in the spirit of the articulatory embeddings used in Dysarthric Speech Recognition Using Curriculum Learning and Articulatory Feature Embedding. That paper uses articulatory features inside the acoustic model; we use them only to decide which unseen confusions are plausible.

Search and false alarms. Edit costs are $-\log P_s$. Search runs over the phone lattice with dynamic programming and deletion penalties, as in A fast fuzzy keyword spotting algorithm based on syllable confusion network. That system used a static population confusion matrix and reported a 34.5% relative gain from its deletion penalty. Fuzzy matching inflates false alarms, so each candidate is scored by a likelihood ratio: the command's score under the personalized costs against its best score under a background of other vocabulary words. This replaces the separately trained false-alarm classifier (a support vector machine) of Word-based probabilistic phonetic retrieval for low-resource spoken term detection. That system fused lattice confidence with unweighted Levenshtein distance and improved ATWV in Assamese from 0.2278 to 0.2451.

Baselines

The naive reference point is uniform-cost phone Levenshtein matching on 1-best output from the same frozen recognizer. This is essentially the CirdoX-style approach of In-home detection of distress calls: the case of aged users, which reached 90.6% F-measure overall but fell to 84.3% for frail users. A second naive point is exact word matching on the output of a strong general-purpose recognizer. Both need no enrollment, so they bound how much personalization adds.

The strongest competitor for enrolled commands is Latent Phrase Matching from Latent Phrase Matching for Dysarthric Speech. It matches enrolled phrase audio templates against incoming speech by dynamic time warping over learned embeddings. It reported about 70% accuracy for severe dysarthria and a 30.5% accuracy improvement on EasyCall. We will reimplement it on the same frozen encoder features with the same enrollment audio. Beating it means matching its accuracy on enrolled commands at small vocabularies and exceeding it at 50 or more commands, where its authors report degradation. On unenrolled commands it simply does not apply.

The strongest acoustic-adaptation competitor is fine-tuning the recognizer's CTC head, plus a residual adapter, on the same $k$ enrollment words, followed by uniform-cost search. It will get the same tuning budget, selected on the same held-out enrollment words. We also re-implement the Morales–Cox per-speaker confusion model with no prior (setting $\alpha_s=0$ with add-one smoothing) on our recognizer. This is the ablation isolating our key ingredient. A second ablation fixes $\pi$ to the pooled population matrix alone, which isolates personalization. For full-data context only, the file reports a 69.4% word accuracy on UASpeech for the CUHK combined system in Development of the CUHK Dysarthric Speech Recognition System for the UA Speech Corpus. That is closed-vocabulary recognition with abundant speaker data, not our task. No reported result was found on Lacuna for few-shot open-vocabulary command spotting on UASpeech or TORGO, so all numbers on our task will come from our own runs.

Why not just…?

Why not just fine-tune the phone recognizer on the enrollment words? This is the real threat. Modern adapters are far more parameter-efficient than the MLLR that lost to metamodels in the Morales–Cox study, so the old gap may not hold. The comparison at $k$ = 5, 20 and 100 settles it directly. We predict a crossover: confusion costs should win at 5–20 words and the gap should close near 100, mirroring the narrowing Morales and Cox saw at 34 sentences. If fine-tuning wins even at 5 words, the core claim is wrong. In that case the proposal reduces to its shortcut-insulation and open-vocabulary contributions, and we will report it that way.

Why not just enroll more templates with Latent Phrase Matching? Templates cannot recognize a command the user never spoke. Their accuracy also drops as the phrase set grows toward 50, by the authors' own account. The vocabulary-scaling and unenrolled-command experiments below test exactly these two limits.

Why not just use a population-level confusion matrix with no personalization? This is cheaper and needs no enrollment. The static matrix of Shao et al. already helped. But the Cirdo-set spread of call error rates across speakers, from 5.3% to 71.4% in Recognition of Distress Calls in Distant Speech Setting: a Preliminary Experiment in a Smart Home, suggests one matrix cannot fit everyone. The population-prior ablation measures this, and we expect its gap to widen with severity.

Why not synthesize the speaker's speech and train on it? Few-shot Dysarthric Speech Recognition with Text-to-Speech Data Augmentation found that recognizers trained only on synthetic speech for unseen speakers from 5 or 100 words had average word error rates in the nineties. We therefore do not rerun this approach as a primary baseline.

Proposed Experiments

Primary spotting task. On UASpeech we build a leave-one-speaker-out spotting task. Prior and pooled confusions come from the other speakers. Enrollment uses $k \in \{5, 10, 20, 100\}$ words from non-evaluation blocks. Evaluation on block 2 uses commands split into enrolled and never-enrolled words, with other words and control speech serving as distractors. Metrics are detection F-measure, equal error rate, and false alarms per hour, reported by intelligibility group. We repeat the task on TORGO, and on EasyCall for comparison with Latent Phrase Matching. On the AD corpus, if available, we test whether personalized costs close the gap between independent (GIR 6) and frail (GIR 2–3) speakers.

Decisive checks. Four results would each settle part of the claim. First, the few-shot crossover against adapter fine-tuning described above. Second, vocabulary scaling from 10 to 200 commands against Latent Phrase Matching. Third, the learned trust $\alpha_s$ should be large for high-intelligibility speakers and yield no loss relative to uniform matching there; this is the Morales–Cox failure we aim to fix. Fourth, a shortcut test following the silence protocol of the UA-Speech/TORGO audit. We run every system on non-speech segments only, and again after re-mixing recordings to equalize signal-to-noise ratio. The proposal predicts that its detections from silence stay at chance, and that its accuracy moves far less under re-mixing than the fine-tuned baselines.