Multi-Intent Commands from Dysarthric Speech Without Intent-Labeled Audio

People with dysarthria, a motor speech disorder caused by conditions such as ALS, cerebral palsy, Parkinson's disease or stroke, depend heavily on voice assistants. These users also tire quickly when they speak. Current voice assistants still expect one goal per utterance, so a request like "turn off the lights and lock the door" either fails or has to be split into two tiring commands. Text-based research has largely solved the compound-request problem on synthetic benchmarks. Dysarthric speech understanding, however, has only been studied for single commands, and nobody will record thousands of annotated compound commands from fatigued clinical speakers.

This note proposes a way to sidestep that annotation bottleneck. The idea is to move the speech into the space where text models already understand compound requests. We train a speech encoder so that, word by word, its outputs land where a frozen text encoder would place the transcript. The only supervision needed is dysarthric audio paired with transcripts, which already exists at scale in the Speech Accessibility Project (SAP). A multi-intent head trained purely on text can then be attached unchanged. That head scores utterances against natural-language descriptions of each intent rather than against arbitrary class IDs. Because the labels are text, a new device command can be added by writing a sentence describing it, without recording anyone. The claim to test is specific. This route needs no intent-labeled dysarthric audio. Its advantage over a transcribe-then-parse cascade should grow with dysarthria severity, because it never commits to a single error-prone transcript.

Core Claim and How It Differs From Prior Work

Two lines of work meet here without overlapping. In text, two papers replace discrete intent IDs with encoded label descriptions. LABAN projects a BERT utterance vector onto the subspace spanned by BERT-encoded intent names and detects unseen intents zero-shot. DGIF injects verbalized slot labels such as "begin artist" through a similar projection and prunes irrelevant intent–token edges, reaching 50.7% overall accuracy on MixATIS and 84.3% on MixSNIPS. Both take clean text as input. The only dysarthric understanding study in the file, A Study into Pre-Training Strategies for Spoken Language Understanding on Dysarthric Speech, decodes acoustic features directly to fixed intent and slot classes on single Dutch home-automation commands (Domotica). It shows that end-to-end training avoids transcription errors, but it uses abstract labels, single intents, and intent-annotated audio.

Our proposal reuses the label-description head from the text work as a method component. Its new contribution is the bridge. A speech encoder is aligned to the frozen text encoder of that head using transcribed dysarthric speech only. The intent and slot supervision comes entirely from text corpora (MixATIS, MixSNIPS). The closest acoustic-side precedent is LaSyn. It goes in the opposite direction, synthesizing speech-encoder latents from text to enlarge typical-speech training data on SLURP with single-intent heads. We instead pull dysarthric speech into text space, so that the text-trained multi-intent reasoning transfers.

Research Program

The pipeline has three pieces.

Acoustic backbone. We start from Parakeet-TDT 1.1B fine-tuned on SAP, following Fine-tuning Parakeet-TDT for Dysarthric Speech Recognition. That work found that freezing parts of the network hurt dysarthric accuracy, so the encoder is fully updated during alignment.

Alignment adapter. A small transformer adapter maps encoder frames into the token space of a frozen BERT text encoder, the one used by the label-description head. We use the recognizer's own frame-to-word alignment to pool frames per word. We then minimize

$$\mathcal{L} = \sum_{i} \left\| g(\bar{h}_i) - e_i \right\|^2 \;-\; \lambda \log \frac{\exp(\langle g(\bar{h}_i), e_i\rangle/\tau)}{\sum_j \exp(\langle g(\bar{h}_i), e_j\rangle/\tau)}.$$

Here $\bar{h}_i$ is the pooled speech representation of word $i$, $g$ is the adapter, $e_i$ is the frozen text-encoder embedding of that word in its transcript context, and $j$ ranges over words in the batch. The weight $\lambda$ balances regression against a contrastive term, and $\tau$ is a temperature. The contrastive term stops the adapter from collapsing confusable words onto an average vector.

Multi-intent head. The head is a DGIF-style label-semantic model, trained once on MixATIS and MixSNIPS text and then frozen. At test time it reads the adapter output in place of BERT token embeddings.

Severity enters in two places. First, alignment training data are stratified by severity, following the Domotica-side finding that including severely impaired speakers helps only when targeting severe users (Pre-Training Strategies). Second, all evaluations are reported per severity band.

Evaluation data. No dysarthric multi-intent benchmark exists; the file reports none. The main test set will be new. Consenting dysarthric speakers will read a few hundred MixSNIPS and MixATIS test utterances in short sessions designed around fatigue, with intelligibility ratings attached. MixATIS was itself built by concatenating single-intent utterances, so we add a second, larger probe built from SAP and TORGO speakers' single-command-like recordings joined with a spoken conjunction. We report this probe separately because it is easier than natural compound speech. We do not use synthetic dysarthric audio for training or testing. Fairness in Dysarthric Speech Synthesis found that F5-TTS clones lose intelligibility at higher severity and worsened mid- and high-severity recognition when added to training.

Baselines

The naive reference is the text-trained DGIF head applied to transcripts from the off-the-shelf whisper-large-v2. The SAP challenge reports 17.82% WER for that recognizer on Test2 (SAP Challenge). This tells us how much the domain gap costs when nothing is adapted.

The strongest competitor is a cascade of a strong recognizer and the same text head. The recognizer is Parakeet-TDT 1.1B fine-tuned on the same SAP training split. It reports 8.11% WER on Test2, and 5.97% WER with 92.53 SemScore on Test1 (Parakeet-TDT). Its transcripts feed the identical frozen DGIF head. This is the right opponent because it uses the same supervision as our method: SAP transcripts plus text intent labels. It also starts from the same backbone and the same text head, so any difference comes only from passing soft aligned vectors instead of a single committed transcript.

We also run an N-best variant of the cascade, which averages head outputs over the top transcripts. This tests whether keeping uncertainty in text form already captures the benefit. All systems get the same SAP data, the same backbone checkpoint, and the same number of adaptation GPU-hours and hyperparameter trials.

Overall multi-intent accuracy on dysarthric audio has no reported result found on Lacuna, so the text ceiling stands in as a reference. That ceiling is DGIF's 50.7% on MixATIS and 84.3% on MixSNIPS on gold text, and we will rerun it on our read prompts' gold transcripts. Beating the cascade means a statistically significant gain in overall accuracy on the moderate and severe bands, with no loss larger than about one point on the mild band.

Two ablations isolate the ingredients. The first swaps the label-description head for a one-hot multi-label head trained on the same text. This tests whether semantic labels matter for transfer or only for adding new commands. The second removes the contrastive term, to check that the adapter is not merely averaging confusable words. A third reference is the Domotica single-intent setting, where we compare against the capsule-network system. That paper reports only relative gains, such as up to 15% absolute slot F1 over its non-pre-trained baseline, so we will reproduce rather than quote its absolute numbers.

Why not just…?

Why not just transcribe with fine-tuned Parakeet and parse the text, perhaps with an LLM? This is the real threat. At 6–8% WER the cascade may match us on mild dysarthria. The evidence that it degrades comes from two places. On typical speech, SLURP shows action accuracy falling from 86.99% to 81.42% at 16.2% WER (SLURP). Parakeet's own authors report that its internal language model deletes repeated phrases, which are exactly the hesitant "and… and" transitions that mark intent boundaries. Whether this matters for compound commands is decided by the severity-stratified comparison and the N-best cascade. If the gap does not widen with severity, we will report that the cascade suffices and that the contribution reduces to adding intents without new recordings.

Why not just use a one-hot multi-label head? It can ride the same alignment bridge, so it might transfer equally well on seen intents. The one-hot ablation tests this directly. What it cannot do is score an intent absent from training. We measure unseen-intent F1 on held-out intents, following LABAN's protocol, where a fixed head scores nothing.

Why not just augment with synthetic dysarthric compound commands? Waveform cloning with F5-TTS degraded mid- and high-severity recognition by 5.74% and 7.65% WER relative to real audio alone (Fairness in Dysarthric Speech Synthesis), and models trained only on synthetic speech failed to generalize. Speed and pitch perturbation (Sapkota et al.) is already part of the backbone recipe, so it is applied to every system rather than offered as an alternative.

Decisive Checks

The idea succeeds if two things hold on the read dysarthric test set. The aligned model's overall accuracy must exceed the Parakeet cascade's on moderate and severe speakers, with the margin growing as intelligibility falls. And the label-description head must retain meaningful unseen-intent F1 where the one-hot head cannot operate. The idea fails, and should be dropped in favor of the cascade, if the N-best cascade closes the gap at every severity level.