Introduction
Dysarthria is a motor speech disorder resulting from neurological injuries such as Cerebral Palsy or Parkinson’s disease. It typically manifests as slurred, slow, or labored speech, creating a significant "acoustic mismatch" for standard Automatic Speech Recognition (ASR) systems. While modern ASR has reached human parity for typical speech, it often fails for dysarthric speakers. Furthermore, while English-centric research is common, there is a critical shortage of assistive technology for Cantonese speakers. Researchers at the Chinese University of Hong Kong (CUHK) have developed specialized multi-modal systems to bridge this gap.
Methodology
The CUHK team proposed two distinct architectures designed to leverage complementary data streams beyond simple audio.
English System: Audio-Visual ASR (AVSR)
For English tasks, the system utilizes visual cues from lip movements to stabilize recognition. The pipeline tracks lip motions using a face alignment network, followed by an autoencoder and Linear Discriminant Analysis (LDA) to extract visual features. These are fused with audio using a Bayesian Gated Neural Network (BGNN). The BGNN dynamically weights the visual input based on the model's uncertainty, accounting for erratic head movements common in dysarthric speakers.
Cantonese System: Prosody Integration
As Cantonese is a tonal language, the system focuses on pitch-related features (Probability of Voicing, Pitch, and Delta-log-Pitch) to compensate for the poor prosodic control often seen in dysarthria. A Gated Neural Network (GNN) integrates these features, preventing uncoordinated pitch "noise" from degrading performance.
Datasets
The systems were evaluated using:
- UASpeech (English): An isolated word recognition task featuring 16 dysarthric speakers.
- CUDYS (Cantonese): A Cantonese dysarthric speech corpus containing word, sentence, and paragraph tasks.
- SpeechOcean: A large-scale normal Cantonese corpus used for data augmentation to overcome the scarcity of disordered speech data.
Key Results
The CUHK systems consistently outperformed both commercial solutions and human listeners. In English tests (UASpeech), the system achieved a 24.0% Word Error Rate (WER) for speaker F04, whereas the Google API and human listeners trailed at 50.6% and 58.1%, respectively. In Cantonese tests (CUDYS), speaker S0006 saw a Character Error Rate (CER) of just 1.7%, a massive improvement over the Google API's 20.6% and the human benchmark of 11.2%.
Conclusion
The success of these systems lies in the "gating" mechanism, which intelligently decides which modality—audio, visual, or pitch—is most reliable at any given moment. While challenges remain regarding data scarcity and speaker dependency, these results demonstrate that multi-modal fusion can provide robust assistive communication tools for individuals with severe speech impairments.
Summary updated