Exploiting Audio-Visual Features with Pretrained AV-HuBERT for Multi-Modal Dysarthric Speech Reconstruction

Xueyuan Chen, Yuejiao Wang, Xixin Wu, Disong Wang, Zhiyong Wu, Xunying Liu, Helen Meng

Researchers from the Chinese University of Hong Kong and Tsinghua University have introduced a pioneering multi-modal framework for Dysarthric Speech Reconstruction (DSR). By integrating both audio and visual cues—specifically lip movements—and leveraging the pretrained AV-HuBERT model, the system achieves a significant breakthrough in assistive technology for individuals with severe speech impairments.

The Challenge of Dysarthric Speech

Dysarthria is a motor speech disorder caused by neurological conditions such as stroke, traumatic brain injury, Parkinson’s disease, or cerebral palsy. It often results in speech that is unintelligible due to distortions in phonemes, timing, and pitch.

Traditional DSR systems rely exclusively on audio signals. However, for patients with severe symptoms, the audio is often too degraded to extract linguistic content reliably. Furthermore, ambient acoustic noise in real-world settings further diminishes the performance of audio-only models. This research addresses this "audio-only bottleneck" by incorporating visual information from the speaker's face, providing a stable, noise-immune source of linguistic data.

Methodology

The proposed system utilizes a multi-modal pipeline designed to decompose dysarthric speech into its constituent parts—content, prosody, and speaker identity—and then reassemble them into clear, natural-sounding speech.

System Architecture

The architecture consists of four primary modules that work in tandem to reconstruct the signal:

  1. Multi-modal Encoder: Extracts phoneme embeddings from fused audio and visual features.
  2. Variance Adaptor: Normalizes the irregular rhythm and pitch typical of dysarthric speech by predicting appropriate duration and pitch for a healthy speaker.
  3. Speaker Encoder: Extracts a d-vector (embedding) from the patient's original speech to preserve their unique vocal identity in the reconstructed output.
  4. Mel-Decoder & Vocoder: Combines the cleaned phonemes, normalized prosody, and speaker identity to generate a mel-spectrogram, which is then converted into a waveform using Parallel WaveGAN.
Overall architecture of the multi-modal DSR system, integrating content, prosody, and speaker identity.

The Multi-modal Encoder Innovation

The core technical contribution is the use of AV-HuBERT (Audio-Visual Hidden Unit BERT). Unlike standard VGG-based fusion models trained from scratch, AV-HuBERT is a self-supervised transformer pretrained on thousands of hours of video. This allows the model to develop a deep understanding of the relationship between lip movements and phonemes, which is critical when the audio signal is severely compromised.

Comparison of the baseline VGG-based encoder and the proposed pretrained AV-HuBERT encoder.

Key Insights

  • Visual Cues as a Structural Skeleton: The study found that as speech severity increases, the relative importance of visual features grows. Even when audio is unintelligible, lip movements often retain the structural "skeleton" of the intended message.
  • The Power of Pretraining: Leveraging AV-HuBERT proved significantly more effective than training multi-modal fusion from scratch on limited dysarthria datasets.
  • Decoupling Content from Prosody: By separating "what was said" (phonemes) from "how it was said" (dysarthric prosody), the system can replace broken prosody with a synthetic, natural version while maintaining the patient's original voice timbre.

Experimental Results

The system was evaluated using the UASpeech benchmark, with additional training data from LJSpeech and VCTK to ensure high-quality prosody and diverse speaker characteristics.

  • Word Error Rate (WER): For the most severe cases (e.g., speaker M12), the system reduced the WER from 98% (nearly unintelligible) to 55.9%, representing a 42.1% absolute improvement.
  • Phoneme Error Rate (PER): The AV-HuBERT encoder reduced phoneme errors by an average of 8.5% compared to audio-only baselines.
  • Naturalness (MOS): Subjective listening tests yielded a Mean Opinion Score (MOS) of approximately 4.0, indicating that the reconstructed speech is perceived as high-quality and natural.

Conclusion and Future Work

While this multi-modal approach sets a new state-of-the-art for DSR, challenges remain. Current high-performing models are speaker-dependent, requiring fine-tuning for specific individuals. Future research will focus on developing "zero-shot" systems that work for any patient immediately and optimizing the AV-HuBERT transformer for real-time communication devices. Additionally, testing the system under more varied, real-world visual conditions will be essential for practical deployment.

Summary updated

BibTeX
@inproceedings{chen2024exploiting,
  author = {Xueyuan Chen and Yuejiao Wang and Xixin Wu and Disong Wang and Zhiyong Wu and Xunying Liu and Helen Meng},
  title = {{Exploiting Audio-Visual Features with Pretrained AV-HuBERT for Multi-Modal Dysarthric Speech Reconstruction}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2024},
  url = {https://openreview.net/forum?id=jmdwKVuYha},
}
Figure