Synthesizing Dysarthric Speech From Damaged Articulators to Rebalance Severity Data

Automatic dysarthria severity classifiers struggle exactly where clinicians need them. There are few recordings of mild-to-moderate and severe speakers, and the benchmark corpora carry a recording artifact that models learn to exploit. On UA-Speech, the dysarthric recordings are about 11 dB noisier than the control recordings. Classifiers reach 96–100% accuracy from the silent gaps between words alone (On using the UA-Speech and TORGO databases…). The usual remedy for scarce classes is to synthesize more examples. Current speech synthesizers clone the voice, and with it the recording channel, of the patient used as a reference. They also break down when asked to imitate severe speech (Data Augmentation using Speech Synthesis…).

We propose a different way to generate minority-class data. Take speech from healthy or mildly affected speakers and convert it into articulator trajectories: time-courses of tongue, lip, and jaw positions. Apply explicit, graded motor impairments in that space, such as shrunken range of motion, slowed and smoothed movement, tremor, and loss of independent tongue-tip control. Then render the result back to audio through a clean decoder and pass every class through the same randomized recording channel. This gives two things acoustic synthesis cannot give together. First, there is a continuous severity knob that can populate the sparse middle of the severity scale. Second, the recording conditions are statistically independent of the label, by construction.

Core Claim

Impairment-parameterized articulatory resynthesis, with channel randomization independent of the label, should do two things at once. It should improve recall on the under-represented intermediate severity classes of UA-Speech, and it should reduce the classifier's reliance on the non-speech shortcut. The strongest acoustic augmentation and rebalancing methods should not match it on both.

The claim has a falsifiable core. If the same impairment parameters applied directly to the audio do as well as applying them to articulator trajectories, then the articulatory model is not earning its place. We will report that outcome as a result.

Positioning Against Prior Work

The closest work is acoustic augmentation for severity classification. Data Augmentation using Speech Synthesis… fine-tunes zero-shot text-to-speech and voice conversion per severity level for a Korean corpus. It found that severe references cause synthesis breakdown, so it generated severe speech from mild references. The authors also did no clinical check that the outputs were pathological. We differ in two ways. Our severity is set by named physical parameters rather than inherited from a reference recording. And because our renderer is clean, it does not copy the channel confound that the audit paper exposed.

Our design borrows its philosophy from Generative Invertible Networks. That work builds virtual TAVR patients by moving along latent axes tied to physical quantities such as calcification. It reports a random forest on its features with 81.55% accuracy and AUC up to 0.88 for paravalvular leakage. We transpose that idea of a "physically meaningful knob" from images to speech. Unlike that work, our knobs are hand-specified from motor physiology rather than discovered.

Two ASR papers supply the articulatory machinery but leave severity aside. Articulatory Features for ASR of Pathological Speech fuses inverted tract variables with filterbanks, giving 19.1% versus 20.3% WER on CHASING01. Exploiting Cross-Domain And Cross-Lingual Ultrasound Tongue Imaging Features… inverts audio to ultrasound features learned on healthy TaL speech, giving a 2.07% absolute WER reduction on TORGO. Both train inversion only on healthy speech, which cannot represent articulation outside the healthy range.

We turn this limitation into a design choice. We only invert healthy and mild speech, which lies inside the training distribution, and we create pathology by forward manipulation. The kinematic speech-coding approach of Coding Speech Through Vocal Tract Kinematics, which represents speech as articulator trajectories that can be decoded back to audio, is our intended encoder–decoder.

The physiology constrains which impairments are plausible. Consideration of muscle co-contraction in a physiological articulatory model shows that independent control of tongue tip and dorsum depends on coordinated co-contraction of opposing muscles. This motivates an impairment that couples tip and dorsum trajectories. The four-cube FEM tongue model shows that intrinsic muscles drive fast blade shaping for /i/. This motivates making high front vowels and coronal consonants degrade first under slowing.

Finally, Speaker-independent dysarthria severity classification… motivates the target. Its speaker-agnostic regularization (SALR) is the strongest UA-Speech result in our file, yet its confusions concentrate between low and medium severity.

Research Program

The generator has three stages. We encode control and mild-severity utterances from the training folds into articulator trajectories. We then apply an impairment operator. For each articulator coordinate $x(t)$, the impaired trajectory is

$$\tilde{x}(t) = c + \alpha\,\big[(h_\tau * x)(t/s) - c\big] + \epsilon\,\sin(2\pi f t),$$

where the terms are defined as follows. $c$ is the speaker's neutral (centroid) posture. $\alpha \le 1$ shrinks range of motion, which produces vowel centralization. $h_\tau$ is a low-pass filter with time constant $\tau$ that limits movement speed. $s \ge 1$ slows the utterance. $\epsilon$ and $f$ add tremor. A separate coupling parameter $\kappa$ mixes tongue-tip and dorsum trajectories to mimic lost independent control. We render $\tilde{x}$ to audio with the clean decoder, then convolve and add noise using recording conditions sampled from one pool shared by all classes.

To tie the parameter vector $(\alpha,\tau,s,\epsilon,\kappa)$ to clinical labels, we fit it per speaker. TORGO includes measured articulatory kinematics, so we can estimate which parameter values take control kinematics closest to each dysarthric speaker's kinematics. We then regress severity on the fitted parameters. Sampling synthetic speakers then means drawing parameters from the fitted conditional distribution for each severity level. This is how we fill the sparse low and medium strata.

Control utterances are also resynthesized and channel-randomized, at $\alpha=1$ and the other parameters set to neutral values. This matters because otherwise "synthetic versus real" could itself become a label cue.

The downstream classifier is the SALR setup itself, a fine-tuned wav2vec 2.0 encoder with the triplet objective. Holding it fixed means any gain comes from the data.

Baselines

The naive reference is the same wav2vec 2.0 classifier trained with plain cross-entropy on the imbalanced folds and no augmentation. It tells us how much any intervention buys.

The strongest existing method on this benchmark is SALR from Speaker-independent dysarthria severity classification…. It reports 70.48% ± 1.11% accuracy and 59.23% ± 1.54% macro F1 under leave-one-subject-out cross-validation on UA-Speech. We will rerun it with its published objective on our exact folds, so the comparison uses identical splits and hyperparameter search budget. Beating it means a macro-F1 gain whose leave-one-subject-out interval does not overlap SALR's. It also requires recall on the low and medium classes to rise without severe-class recall falling.

The strongest augmentation competitor is per-severity xTTS and voice-conversion self-augmentation following Data Augmentation using Speech Synthesis…. That paper reports no global accuracy figure in our file; no reported result found on Lacuna for UA-Speech accuracy with that method. We will match the number of synthetic utterances and sweep the same augmentation ratios, including the 1:1 ratio it found best.

A rebalancing baseline adds class-weighted loss and minority oversampling to SALR. This tests whether more data matters beyond reweighting.

A channel-equalization baseline adds noise to control recordings to match the dysarthric SNR distribution.

Three ablations isolate the key ingredient. In the first, resynthesis is used without impairment. This isolates the effect of channel randomization alone. In the second, the same fitted parameters are applied in acoustic space to the clean resynthesized audio: time-stretching for $s$, formant warping toward the centroid for $\alpha$ using area-function-to-formant relations in the spirit of Mappings between vocal tract area functions, vocal tract resonances and speech formants…, and amplitude–frequency modulation for tremor. This isolates articulatory-space manipulation from the knob values themselves. In the third, channel conditions are correlated with the label as in the real data, which shows whether decoupling matters.

Why not just…?

Why not just use zero-shot TTS or voice conversion conditioned on patients? These methods condition on patient audio, so they inherit that audio's channel. At the 11 dB gap documented by the audit paper, that would reinforce the shortcut. The same methods also fail on severe references (Data Augmentation using Speech Synthesis…). This is still a genuine threat on raw accuracy, because cloned voices are more natural than decoded kinematics. Our silence test and intermediate-class recall decide the matter: if voice conversion matches us on both, our method is not needed.

Why not just reweight the loss or oversample? Neither adds new acoustic evidence to the sparse classes. When the minority classes are also the noisy-channel classes, up-weighting them rewards the background-noise cue. The rebalancing baseline, scored on non-speech-segment accuracy, measures this directly.

Why not just equalize the recording channel by adding noise to controls? This is the cheapest fix for the shortcut, and we expect it to cut silence-test accuracy. But it adds no intermediate-severity examples. It also cannot remove channel cues beyond SNR, such as microphone and room colouration. We run it alone and combined with our generator, so it is possible to see whether channel matching plus reweighting already explains the gains.

Why not just perturb the audio with speed changes and formant warping? This is the most serious alternative, because slowing and vowel centralization are major acoustic signs of dysarthria. The second ablation answers it at matched parameter values. Articulatory manipulation should help only where impairment effects are coupled across sounds, such as consonant imprecision arising from tip–dorsum coupling. If the acoustic-space version ties, we will conclude the gain lies in the parameterization and channel decoupling, not in the vocal tract model.

Proposed Experiments

The primary benchmark is UA-Speech with four-level severity under leave-one-subject-out cross-validation. TORGO serves as the external test and as the kinematic calibration source; calibration and evaluation speakers are kept disjoint. Metrics are as follows:

  • macro F1;
  • per-class recall, with emphasis on low and medium;
  • ordinal mean absolute error in severity levels;
  • the speaker-identity linear probe from the SALR paper;
  • a shortcut score: accuracy when the trained model receives only the voice-activity-detected non-speech segments, where chance is 25%.

A further channel-swap test adds dysarthric-level noise to held-out control speech and counts how many predictions move toward dysarthric labels.

The decisive checks, in order of importance, are the following. The first is that the full method beats rerun SALR on macro F1 and low/medium recall while lowering the shortcut score toward chance. The second is that kinematic-space impairment beats the acoustic-space ablation at matched parameters. If it does not, the articulatory claim fails. The third is that fitted impairment parameters on TORGO's measured kinematics vary monotonically with clinical severity, which validates the knob. The fourth is that a blinded speech-language therapist rates synthetic samples at their intended severity more often than xTTS samples. This fourth check is the clinical validity test that the prior augmentation work explicitly lacked.

A known risk is that the decoder renders extreme impaired trajectories poorly, since it was trained on healthy speech. We will report decoder reconstruction quality as a function of $\alpha$ and $\tau$, and cap sampling where quality collapses.