Speech recognizers fail badly on dysarthric speech, the slurred, effortful, unevenly loud speech produced by people with conditions such as cerebral palsy, ALS, or Parkinson's disease. Real recordings are scarce because producing them tires the speaker, so researchers generate synthetic dysarthric speech with text-to-speech (TTS) models and add it to training data. The synthetic speech has a consistent flaw. It copies the speaker's timbre almost perfectly but smooths out the moment-to-moment instability in loudness and pitch that defines the disorder. As a result it helps mildly impaired speakers and hurts severely impaired ones.
We propose to treat this as a dynamic-range problem and to borrow the tool built for it. Dynamic range compression (DRC) is the audio technique that reshapes how a signal's loudness envelope maps from input to output. The speech-in-noise literature has spent years using it to flatten speech so that weak consonants survive background noise. We run the operation in reverse. For each target speaker we fit a small, parametric "dynamics profile": an expansion curve, attack and release times, and a matched envelope modulation. We fit it from a handful of that speaker's real utterances, then apply it to TTS output so that the synthetic speech inherits the speaker's measured loudness structure. The profile has about ten parameters, so it can be fitted from a few seconds of audio. That matters because fine-tuning the TTS model itself on unseen dysarthric speakers has been shown to collapse.
Core Claim
The gap between synthetic and real dysarthric speech is concentrated in the temporal structure of the energy envelope and the pitch contour, not in the spectral envelope. We claim that a low-dimensional, speaker-fitted transform of that structure closes more of the gap for mid- and high-severity speakers than three alternatives: unstructured loudness jitter, more synthetic data, or a larger recognizer.
The claim rests on three findings in the file. Synthetic Dysarthric Speech: A Supplement, Not a Substitute fine-tuned BERT-VITS2 on the Chinese Dysarthric Speech Database (CDSD). It found MFCC cosine similarity above 0.998 to real speech, yet the synthetic speech lacked F0 instability and loudness fluctuation, and scaling synthetic volume to 7–9x gave only marginal gains. Training Data Augmentation … by Text-to-Dysarthric-Speech Synthesis trained Grad-TTS from scratch on TORGO and reports that synthetic speech sounds systematically less severe than the real recordings. Fairness in Dysarthric Speech Synthesis cloned voices zero-shot with F5-TTS. Speaker similarity and prosody scores showed little bias, but intelligibility showed large disparate impact at high severity (disparate impact of 0.59 on average word error rate), and augmentation hurt mid- and high-severity recognition.
These three papers motivate the proposal: each documents the smoothing, and none attempts a signal-level correction. The method comes from the DRC literature. The input-output envelope curve of Speech-in-noise intelligibility improvement based on spectral shaping and dynamic range compression and the soft-knee level detector of Speech synthesis enhancement in noisy environments give us a parameterization whose behavior is well understood. We use it for expansion rather than compression.
Research Program
The transform acts on the log envelope $e(t)$ of a synthetic utterance, measured in dB per frame:
$$\hat{e}(t) = g_\theta\big(\bar{e}_{\tau_a,\tau_r}(t)\big) + \big(e(t) - \bar{e}_{\tau_a,\tau_r}(t)\big) + \sigma\, m_\phi(t)$$
The symbols are defined as follows. $\bar{e}_{\tau_a,\tau_r}$ is the envelope smoothed by a level detector with attack time $\tau_a$ and release time $\tau_r$. $g_\theta$ is a piecewise-linear input-output curve whose segment slopes may exceed one, which means expansion. $m_\phi(t)$ is a unit-variance noise process whose modulation spectrum is shaped by a few parameters $\phi$, and $\sigma$ scales it. The middle term keeps the fine phonetic detail, so only the slow loudness structure is rewritten.
The gain $\hat{e}-e$ is applied to the waveform. A matching second channel perturbs F0 through a vocoder, with fitted jitter magnitude and modulation rate, because the file identifies pitch instability as the other missing dynamic.
The parameters are fitted per speaker. We minimize a sliced Wasserstein distance between feature distributions from the speaker's real recordings and from transformed synthetic renderings of the same text. The features are frame-energy histograms, envelope modulation spectra, consonant-to-vowel energy ratios, and F0 standard deviation. The fit uses gradient-free search, since the parameter space is small.
The program has three stages. The first measures the gap: compute these statistics for real and Grad-TTS, BERT-VITS2 and F5-TTS speech, stratified by severity, and check that the envelope and F0 discrepancies grow with severity while spectral discrepancy does not. The second fits profiles from 1, 5 and 20 real utterances per speaker and measures how much of the gap closes on held-out utterances. The third uses the transformed speech as augmentation for Whisper and HuBERT recognizers.
Background noise enters as an evaluation condition, not a design component. Hypokinetic dysarthria already lowers consonant energy (Moro-Velazquez et al.), and noise masks low-energy segments first (Modeling Noise Influence … by Reduced Speech Dynamic Range). A recognizer trained on over-smoothed synthetic speech should therefore be especially fragile in babble. We test that by adding cafeteria babble at several signal-to-noise ratios (SNRs).
Baselines
The naive reference is the recognizer fine-tuned on authentic data only. On CDSD this is reported as 35.1% character error rate (CER) by Synthetic Dysarthric Speech. On TORGO under leave-one-speaker-out evaluation, the unadapted Whisper-Medium figure is 56.14% word error rate (WER) in Training Data Augmentation ….
The strongest existing augmentation method on TORGO is that paper's Grad-TTS pipeline. It trains a text-to-dysarthric-speech model from scratch and fine-tunes Whisper-Large with the synthetic data plus SpecAugment, reporting 16.68% WER. We reproduce it with the same splits, the same synthetic hours and the same fine-tuning schedule. Our only change is to pass its synthetic output through the fitted profile. Beating it means a lower overall WER with no increase at low severity, and a statistically reliable reduction on high-severity held-out speakers.
On CDSD the strongest comparison is authentic plus BERT-VITS2 supplement at 33.1% CER, from the same paper. We re-run it with and without the profile at equal synthetic volume.
On UA-Speech the strongest systems are the Spectral Basis GAN augmentation and multi-system combination of Enhancing Pre-trained ASR System Fine-tuning … Adversarial Data Augmentation, at 16.53% WER overall and 46.47% on very-low-intelligibility speakers. We compare against its single-model Wav2vec2.0 setting, which gained 2.01% absolute over speed perturbation, so the comparison is not confounded by system combination.
Two ablations isolate the key ingredient, and they are the most important comparisons. The first applies random envelope and F0 jitter with variance matched to the fitted profile but no temporal structure and no speaker fit. The second applies a profile fitted to a different speaker of the same severity. If speaker-specific temporal structure matters, the fitted profile must beat both. All arms use identical synthetic text, hours, tuning budgets, and three training seeds.
Why not just…?
Why not just add random gain and loudness jitter? This is the most serious threat. Standard volume perturbation is cheap, and the recognizer might only need robustness to loudness variation, not realistic variation. The file contains no result on it for dysarthric speech. The matched-variance random-jitter ablation is designed to settle this. If it performs within seed noise of the fitted profile, we would report that the useful ingredient is variance alone and drop the claim about speaker-specific structure.
Why not just generate more synthetic data or use a stronger TTS model? Synthetic Dysarthric Speech found that 7–9x more synthetic data gave only marginal gains. Fairness in Dysarthric Speech Synthesis shows that a large zero-shot cloner still harms severe speakers. Both point to a systematic bias that scale does not remove. We also run a doubled-volume, unprofiled arm to confirm this in our setup.
Why not just fine-tune the TTS model on each new speaker? Few-shot Dysarthric Speech Recognition with TTS Data Augmentation reports average WERs in the nineties when synthetic speech for unseen speakers drives recognition. Few-shot adaptation of a full generator is the failure mode our low-dimensional profile is meant to avoid. We compare against per-speaker TTS fine-tuning at equal real-audio budgets of 1, 5 and 20 utterances.
Why not just use SpecAugment or speed perturbation? Speed perturbation was beaten by GAN augmentation in Enhancing Pre-trained ASR …. Duration morphing alone slightly degraded word accuracy (28.2% to 27.3%) in Acoustic transformations to improve the intelligibility of dysarthric speech. SpecAugment masks information but does not add the missing dynamics. We keep SpecAugment in every arm, so any gain from the profile is additive to it.
Proposed Experiments
The first decisive check comes before any recognizer is trained. It asks whether the gap is really in the dynamics. If envelope and F0 statistics of synthetic speech do not diverge from real speech more at high severity than at low severity, the premise fails and we stop.
The second check is a transfer test. We fit profiles on dysarthric speakers and apply them to healthy control speech from TORGO. A severity classifier trained on real data should then shift its predictions toward the source speaker's severity. If it does not, the profile captures nothing perceptually or diagnostically meaningful.
The third check is the recognition comparison described under Baselines. It covers TORGO leave-one-speaker-out, CDSD's seven-speaker speaker-independent subset, and UA-Speech unseen-speaker few-shot runs, reported by severity group, because the claim is specifically about mid and high severity.
The fourth check repeats the TORGO and UA-Speech test sets mixed with cafeteria babble at 10, 5 and 0 dB SNR. We found no reported result on Lacuna for dysarthric ASR under additive noise, so the unprofiled Grad-TTS arm defines the reference curve. We predict that the profiled arm's advantage widens as SNR falls.
As an interpretive check, we examine error types. If the gains come from fewer consonant deletions, it would link the result to the energy-reallocation account from the noise literature. Human listening tests are out of scope for the recognition claim, and we do not report naturalness.
References
- Synthetic Dysarthric Speech: A Supplement, Not a Substitute for Authentic Data in Dysarthric Speech Recognition
- Training Data Augmentation for Dysarthric Automatic Speech Recognition by Text-to-Dysarthric-Speech Synthesis
- Fairness in Dysarthric Speech Synthesis: Understanding Intrinsic Bias in Dysarthric Speech Cloning using F5-TTS
- Speech-in-noise intelligibility improvement based on spectral shaping and dynamic range compression
- Speech synthesis enhancement in noisy environments
- Study of the Performance of Automatic Speech Recognition Systems in Speakers with Parkinson's Disease
- Modeling Noise Influence to Speech Intelligibility Non-Intrusively by Reduced Speech Dynamic Range
- Enhancing Pre-Trained ASR System Fine-Tuning for Dysarthric Speech Recognition Using Adversarial Data Augmentation
- Few-shot Dysarthric Speech Recognition with Text-to-Speech Data Augmentation
- Acoustic transformations to improve the intelligibility of dysarthric speech