Dysarthric speech reconstruction takes speech from a person whose motor control of articulation is impaired and turns it into clearer speech for a listener, ideally in the speaker's own voice. These systems are scored with two kinds of machine metric. One is intelligibility: the word error rate (WER) an automatic speech recognizer gets when it transcribes the reconstructed audio. The other is naturalness: a neural predictor of mean opinion score (MOS), which estimates how human raters would score the audio. Both metrics reward a failure mode that is dangerous in a communication aid, which is a crisp, natural-sounding word that is not the word the speaker said. A garbled word tells the listener to ask again. A fluent wrong word misleads them and gives no warning. Machine WER counts the two errors identically, and MOS predictors actively prefer the fluent one.
We propose to measure this divergence directly and to design against it. The test is a human listening study that records not only which word a listener heard but how confident they were. From it we compute a "confident error rate": the fraction of words heard confidently that were wrong. The method makes the reconstruction's clarity track how sure the system is about the content. A speaker-adapted recognizer supplies calibrated, frame-by-frame confidence, and that confidence sets how strongly a generative refiner is allowed to pull each stretch of audio toward clean, canonical speech. Where the system is unsure, the output stays close to the speaker's original acoustics. The claim is that this reduces confident listener errors at little cost in correct recognition, and that standard metrics cannot see the difference.
Core Claim
The perceptual-objective gap for atypical speech is not mainly the one documented for codecs and enhancement, where reference metrics penalize harmless phase shifts (ViSQOL v3) or plausible acoustic detail (A Unified Neural Architecture for Instrumental Audio Tasks). It is a gap about content. A generative restorer that fixes the sound can also swap the word, and neither machine WER nor predicted naturalness distinguishes a swap the listener can detect from one they cannot. We make three claims. First, across reconstruction systems, machine WER and predicted MOS will correlate weakly or even negatively with the human confident error rate. Second, making restoration strength conditional on the adapted recognizer's calibrated confidence lowers the confident error rate compared with any single global restoration strength at matched human correct-word rate. Third, this requires the recognizer to be speaker-adapted, because an unadapted recognizer's confidence on dysarthric speech is too poorly calibrated to be useful as a gate.
Where This Sits in Prior Work
Two results motivate the proposal. Diffiner refines the output of enhancement and separation systems using a diffusion model, a generator that learns by gradually denoising random noise into samples, trained on clean speech. It shows that reference-free quality scores and human MUSHRA listening scores rise while signal-to-distortion ratio stays flat or drops. It exposes a single global blending parameter $\alpha$ to trade fidelity against perceived quality. We reuse its refinement mechanism, but we make $\alpha$ vary over time and depend on content confidence, and we apply it to disordered speech, which the paper does not consider. Ma et al. show that generative target-speaker extraction sounds better than masking but hallucinates or alters content, which they counter with a joint recognition loss. Their fix optimizes machine intelligibility. We argue that machine intelligibility is exactly the metric that hides fluent substitutions, so we add a human confidence-aware measure and a gating mechanism rather than another recognition loss.
The closest application work is Adversarial Speaker Adaptation. It adapts a speaker encoder to dysarthric speakers and uses an adversarial regularizer so reconstructed speech keeps the speaker's identity while becoming intelligible. It reports 22.3% and 31.5% absolute WER reductions on UASpeech for moderate and moderate-severe dysarthria. Those numbers come from a machine recognizer on reconstructed audio, which is the metric we expect to overstate benefit when substitutions are fluent. We use this system as a baseline and re-score it under the human measure.
Two works supply the adaptation machinery and the confidence source. Structured Speaker-Deficiency Adaptation adds adapters that separate speaker identity from severity in speech foundation models and reaches 19.45% WER on UASpeech, with 49.34% on the very-low-intelligibility group. We use it as the adapted recognizer whose frame-level posteriors gate restoration. Personalized Adversarial Data Augmentation supplies synthetic dysarthric speech that we use to enlarge the calibration set.
Preference-alignment work constrains the design. Aligning Generative Speech Enhancement with Perceptual Feedback uses Direct Preference Optimization, which fine-tunes a model directly on pairs of preferred and rejected outputs, with UTMOS rewards and reports up to 56% relative quality gains on DNS 2020. SpeechJudge trains a generative reward model that reaches 77.2% agreement with human naturalness preferences, against under 70% for Gemini-2.5-Flash. Both reward naturalness, and both are trained on typical speech. We treat them as the mechanism most likely to increase fluent substitutions if applied naively to dysarthric reconstruction, and we test that directly instead of adopting them. Jia et al. show that human perceptual labels from synthesis evaluation transfer to dysarthria severity assessment, which suggests perceptual supervision can cross into this domain. However, they score impairment and do not reconstruct speech.
Research Program
The system has three parts. The first is a speaker-adapted recognizer built with the structured adapters of Hu et al. It is trained on UASpeech and TORGO training speakers and run on each test utterance to produce frame-level phone posteriors. The second is a restorer: a Diffiner-style diffusion refiner that operates on the dysarthric waveform itself, so its output stays time-aligned with the input, and that is optionally conditioned on the recognized phone sequence. We fine-tune the clean-speech prior briefly on each speaker's control-condition or least-impaired recordings so that restored segments keep their voice. The third part is the gate. It converts calibrated confidence $c_t$ at frame $t$ into a restoration strength
$$\alpha_t = \sigma\big(k\,(c_t - \tau)\big),$$
where $\sigma$ is the logistic function, $\tau$ is a confidence threshold, and $k$ sets how sharp the transition is. Here $\alpha_t$ plays the role of Diffiner's blending weight at frame $t$: near 1 the frame follows the phone-conditioned clean prior, and near 0 it stays close to the observed signal. We fit calibration of $c_t$ by temperature scaling on held-out speakers and check it with reliability curves broken down by severity group. The parameters $\tau$ and $k$ are tuned on a development set against the human-derived objective described below. To keep that tuning affordable, we use a proxy fitted on a pilot listening study.
The analysis side is the other half of the contribution. For every system we collect human transcriptions of reconstructed isolated words. UASpeech prompts are known, so correctness is unambiguous. Listeners also give a three-level confidence rating. We define confident error rate as the share of high-confidence responses that are wrong, and correct rate as the share of all responses that are right. We then ask how well machine WER from an off-the-shelf recognizer, UTMOS, DNSMOS, SpeechJudge-GRM preference, and speaker similarity each rank systems relative to these two human quantities. This yields the divergence measurement that the gap analysis says is missing for atypical speech.
Baselines
The naive reference point is the unprocessed dysarthric recording played to listeners. It has no substitutions introduced by processing, so its confident error rate is the floor that any reconstruction has to justify moving away from.
The strongest existing reconstruction system in the file is Adversarial Speaker Adaptation (Wang et al.). Its reported result is 22.3% absolute machine-WER reduction for moderate and 31.5% for moderate-severe dysarthria on UASpeech. We retrain it on the same speaker splits and data as our system. We report its machine WER to confirm we reproduce the direction of its reported gains, then score it with the human protocol. Beating it means a lower confident error rate at equal or higher human correct rate, and we do not need to beat it on machine WER.
The strongest competitor on the specific mechanism is the same Diffiner-style refiner with a single global $\alpha$, swept over a grid (Diffiner). It traces a trade-off curve between correct rate and confident error rate. It gets the same restorer, the same per-speaker prior fine-tuning, and a tuning grid at least as large as our search over $\tau$ and $k$. Our method wins only if its operating points lie below that curve, meaning fewer confident errors at the same correct rate.
We run two ablations to isolate the key ingredient. The first replaces adapted confidence with confidence from the unadapted foundation model, which tests whether adaptation is what makes the gate work. The second replaces $c_t$ with a random signal matched to the same overall mean restoration strength, which tests whether the gain comes from where restoration is applied rather than how much. For the recognizer itself, we reproduce the adapters of Hu et al., whose reported UASpeech WER is 19.45%, so that the confidence source matches the strongest adaptation result in the literature.
Why not just…?
Why not just recognize the word with the adapted model and resynthesize it with a personalized text-to-speech voice, passing the original audio through when confidence is low? This hard-threshold cascade is the most serious threat, especially on isolated-word UASpeech, where recognizing a closed vocabulary is easy. We include it as a baseline with the same recognizer and a threshold tuned on the same budget. It is the limit of our gate as $k$ goes to infinity with restoration replaced by synthesis. If it matches our operating curve, the honest conclusion is that graded, time-aligned restoration buys nothing on isolated words. In that case the contribution narrows to the evaluation protocol plus the TORGO sentence results, where partial-confidence words are common and hard switching produces audible splices. The experiment settles the question either way.
Why not just align the reconstruction model with a perceptual reward, using preference optimization with UTMOS or SpeechJudge-GRM? The rewards in Li et al. and SpeechJudge score naturalness and were built on typical speech. Our hypothesis is that they reward fluent substitutions. We run preference optimization on the global-$\alpha$ refiner with each reward and predict that predicted MOS rises while confident error rate also rises. If confident errors instead fall, the hypothesis is wrong, and reward alignment becomes the simpler recommended route.
Why not just fix the machine metric, for example by scoring reconstructions with the adapted recognizer's confidence-weighted WER, and skip human studies? That metric uses the same model as the gate, so it would be circular. Whether any machine proxy tracks human confident errors is precisely the empirical question here. We fit such proxies on part of the listening data and test their ranking agreement on held-out systems. If one generalizes, it becomes a cheap evaluation tool, which is a useful outcome in its own right.
Proposed Experiments and Decisive Checks
We will run the experiments on UASpeech, which offers isolated words, known prompts, and four intelligibility groups, and on TORGO, which offers sentences with dysarthria from cerebral palsy or ALS. Test speakers are unseen during restorer training. Only the adapter and prior fine-tuning use a speaker's own adaptation split. The machine metrics are WER from an off-the-shelf recognizer and from the adapted recognizer, UTMOS, DNSMOS, SpeechJudge-GRM pairwise preference, and speaker-embedding similarity. The human metrics are correct rate and confident error rate, collected from naive listeners with a fixed number of responses per word per system, randomized system order, and bootstrap confidence intervals over speakers. No reported result was found on Lacuna for any human listener-confidence or undetected-error measure on these corpora, so all human figures will be new.
The first decisive check is the divergence itself. Across roughly eight systems (unprocessed, Adversarial Speaker Adaptation, global-$\alpha$ grid points, preference-aligned variants, the cascade, and ours), we compute the rank correlation of each machine metric with confident error rate. We expect it to be weak or of the wrong sign for machine WER and MOS predictors, especially in the very-low-intelligibility group. The second check is the operating-curve comparison against global $\alpha$ and the cascade, reported per severity group. The central prediction is that the gain is largest for severe speakers, where recognizer confidence is most variable. The third check is the pair of ablations: if unadapted confidence or random gating performs as well, the adaptation-driven gate is not the cause. A final calibration check confirms the premise. If the adapted recognizer's reliability curves on held-out severe speakers are no better than the unadapted model's, the third claim fails, and we will report that.
References
- ViSQOL v3: An Open Source Production Ready Objective Speech and Audio Metric
- A Unified Neural Architecture for Instrumental Audio Tasks
- Diffusion-based Signal Refiner for Speech Enhancement and Separation
- Enhancing Intelligibility for Generative Target Speech Extraction via Joint Optimization with Target Speaker ASR
- Speaker Identity Preservation in Dysarthric Speech Reconstruction by Adversarial Speaker Adaptation
- Structured Speaker-Deficiency Adaptation of Foundation Models for Dysarthric and Elderly Speech Recognition
- Personalized Adversarial Data Augmentation for Dysarthric and Elderly Speech Recognition
- Aligning Generative Speech Enhancement with Perceptual Feedback
- SpeechJudge: Towards Human-Level Judgment for Speech Naturalness
- Augmenting Open-Vocabulary Dysarthric Speech Assessment with Human Perceptual Supervision