Robust Speech Recognition for Disordered and Atypical Speech

40 concepts39 papers1 proposal

Modern Automatic Speech Recognition (ASR) systems achieve near-human accuracy on standard, healthy adult speech. However, their performance degrades sharply when encountering atypical or disordered speech—such as dysarthria caused by neuro-motor conditions like ALS or cerebral palsy, aphasia, or the natural acoustic variations found in early childhood and elderly populations.

As highlighted by Zheng et al. in The Interspeech 2025 Speech Accessibility Project Challenge, the primary bottleneck in this domain is extreme data scarcity. While standard models train on tens of thousands of hours of data, clinical speech corpora are tiny because recording audio is physically exhausting for patients and labor-intensive for experts to annotate.

To bridge this gap, researchers do not simply wait for more data. Instead, they use three core strategies to make models robust to atypical speech: generative data augmentation to simulate pathological speech, articulatory and spectro-temporal priors to guide feature extraction, and cross-domain transfer learning to adapt healthy speech models to atypical patterns.

Disentangling Content and Pathology via Generative Augmentation

If real pathological data is scarce, a natural solution is to synthesize it. However, standard Text-to-Speech (TTS) models struggle to capture the highly irregular, non-linear acoustic signatures of conditions like dysarthria. As Li et al. demonstrate in Synthetic Dysarthric Speech: A Supplement, Not a Substitute for Authentic Data in Dysarthric Speech Recognition, purely synthetic data cannot yet fully replace authentic clinical recordings because it fails to capture the full spectrum of organic speech degradation.

To make synthetic data effective, generative models must separate what is being said (linguistic content) from how it is impaired (pathological traits). Jin et al. address this in Adversarial Data Augmentation Using VAE-GAN for Disordered Speech Recognition by building a VAE-GAN framework that explicitly disentangles content from speaker impairment traits. This allows the system to project the acoustic characteristics of dysarthria onto healthy speech templates, generating diverse training samples from abundant healthy data.

[Healthy Speech Content] ---\
                             ---> [VAE-GAN Disentanglement] ---> [Synthetic Pathological Speech]
[Pathology Acoustic Trait] --/

Similarly, Jin demonstrates in Adversarial Data Augmentation for Disordered Speech Recognition that Deep Convolutional Generative Adversarial Networks (DCGANs) can map fine-grained articulatory imprecisions onto normal speech. To optimize how these synthetic variations are fed into an ASR system, Jin et al. later introduced a reinforcement learning framework in Towards Automatic Data Augmentation for Disordered Speech Recognition to dynamically discover optimal augmentation policies rather than relying on handcrafted parameters.

Beyond GANs, modern generative architectures like diffusion models and large language models (LLMs) are pushing these boundaries further. In DiffDSR: Dysarthric Speech Reconstruction Using Latent Diffusion Model, Chen et al. utilize a Latent Diffusion Model paired with self-supervised foundation models to reconstruct impaired speech into clear audio while preserving the speaker's identity. For personalization, Wagner et al. show in Personalized Fine-Tuning with Controllable Synthetic Speech from LLM-Generated Transcripts for Dysarthric Speech Recognition how combining parameter-efficient fine-tuning (AdaLoRA) with LLM-generated transcripts can create highly controllable synthetic pipelines tailored to individual speakers. To address voice pathology classification, Koudounas demonstrates in Voice Disorder Analysis: a Transformer-based Approach that synthetic data generation can be successfully paired with a Mixture of Experts architecture to analyze both sustained vowels and continuous sentence readings.

Anchoring Models with Articulatory and Spectro-Temporal Priors

Standard acoustic features like Mel-Frequency Cepstral Coefficients (MFCCs) or raw frame embeddings are designed to optimize for linguistic content while treating subtle acoustic anomalies as noise. For disordered speech, however, those "anomalies" are the defining characteristics of the speaker's articulation. When data is scarce, large-scale pre-trained models often struggle because they are optimized to ignore these variations. As Balagopalan et al. point out in Comparing Pre-trained and Feature-Based Models for Prediction of Alzheimer's Disease Based on Speech, traditional feature engineering based on clinical domain expertise often remains highly competitive with deep transformer models in low-data regimes.

To prevent models from misinterpreting these patterns, researchers inject physical and acoustic priors into the feature extraction process. For instance, Wong et al. in Analysis of Dysarthric Speech using Distinctive Feature Recognition use Multi-Layer Perceptrons to map speech sounds to "Distinctive Features" (DFs)—the fundamental phonological building blocks of human speech (such as nasality or voicing). By measuring deviations from canonical, healthy speech at the feature level, the model anchors its predictions on where articulation physically breaks down.

Taking a mathematical approach to feature separation, Geng et al. introduce "spectro-temporal deep features" in Spectro-Temporal Deep Features for Disordered Speech Assessment and Recognition and Speaker Adaptation Using Spectro-Temporal Deep Features for Dysarthric and Elderly Speech Recognition. By applying Singular Value Decomposition (SVD) to acoustic representations, they isolate static spectral shapes (the intended phonemes) from dynamic temporal variations (slurred pacing or irregular speaking rates). This allows the downstream neural network to adapt to speaker-specific motor constraints without losing track of the underlying words.

Raw Audio ---> Spectrogram ---> Singular Value Decomposition (SVD)
                                       |
                                       ├──> Spectral Components (What phoneme is intended)
                                       └──> Temporal Components (How pacing/timing is impaired)

In cases where audio alone is insufficient, researchers look directly at the physical mechanics of speech or use aggressive data manipulation. Hu et al. demonstrate in Exploiting Cross-Domain And Cross-Lingual Ultrasound Tongue Imaging Features For Elderly And Dysarthric Speech Recognition that integrating tongue movements estimated from ultrasound imaging provides an invariant articulatory anchor. This cross-modal grounding allows models to transfer structural speech knowledge across entirely different languages and domains. To ensure these representations remain stable across environments, Liss et al. propose a framework in Robust Vocal Quality Feature Embeddings for Dysphonic Voice Detection that uses contrastive learning paired with aggressive "data warping" (simulated noise and distortion) to extract robust digital signatures of voice quality that generalize across recording environments.

Cross-Domain Adaptation and Multi-Task Learning

Because training a deep learning model from scratch on low-resource clinical data inevitably leads to overfitting, the field relies heavily on transferring knowledge from high-resource domains (like healthy adult speech) to the target demographic.

Early efforts, such as the system developed by Yu et al. in Development of the CUHK Dysarthric Speech Recognition System for the UA Speech Corpus, proved that combining advanced neural architectures (TDNNs and LSTMs) with explicit cross-domain knowledge from normal speech drastically reduces word error rates. To handle the statistical uncertainty inherent in small datasets, Deng et al. propose a dual approach in Bayesian Parametric and Architectural Domain Adaptation of LF-MMI Trained TDNNs for Elderly and Dysarthric Speech Recognition. They use a Bayesian framework to manage parameter uncertainty alongside Differentiable Architecture Search (DARTS) to dynamically optimize the model's temporal context window for irregular speech rates.

Knowledge transfer can also be achieved through architectural constraints and distillation. Wang et al. introduce a staged knowledge distillation framework in Staged Knowledge Distillation for End-to-End Dysarthric Speech Recognition and Speech Attribute Transcription to pass down robust representations from models trained on healthy speech to smaller, specialized networks. For highly specialized tasks like wake-up word spotting, Liu et al. demonstrate in Optimizing Dysarthria Wake-Up Word Spotting: An End-to-End Approach for SLT 2024 LRDWWS Challenge that combining multi-task learning with a dual-filter verification strategy prevents false activations caused by the high acoustic variability of impaired speech.

Multi-task learning offers an alternative way to enforce shared representations. Qin demonstrates in Automatic Speech Assessment for People with Aphasia Using TDNN-BLSTM with Multi-Task Learning that co-training an acoustic model on both large-scale unimpaired speech and aphasic speech forces the network to learn robust, generalizable text features.

This strategy is equally effective across demographic boundaries where data is scarce. For example, Chen et al. show in Multi-Task Learning for Mispronunciation Detection on Singapore Children's Mandarin Speech that sharing hidden layers between adult and child speech models mitigates the chronic data scarcity and high acoustic variability typical of developing voices. This aligns with findings from Zhou et al. in ChildMandarin: A Comprehensive Mandarin Speech Dataset for Young Children Aged 3-5, which notes that early childhood speech datasets necessitate a heavy reliance on fine-tuning and shared representations rather than large-scale foundational training from scratch.

Key Foundational Work

Automatic dysfluency detection in dysarthric speech using deep belief networks (2015) — contributed 1 concept elements to this direction.
Polynomial dynamic time warping kernel support vector machines for dysarthric speech recognition with sparse training data (2005) — contributed 1 concept elements to this direction.
Analysis of Dysarthric Speech using Distinctive Feature Recognition (2015) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_low_resource_speech_recognition_f2ff51ed,
  title = {low-resource speech recognition},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/low-resource-speech-recognition/rdr_f2ff51ed4a8d403283f5b4437cbc2f90},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_low_resource_speech_recognition_f2ff51ed_v1,
  title = {low-resource speech recognition},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/low-resource-speech-recognition/rdr_f2ff51ed4a8d403283f5b4437cbc2f90/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake