Personalized One-Shot Lipreading for an ALS Patient

Bipasha Sen, Aditya Agarwal, Rudrabha Mukhopadhyay, Vinay P. Namboodiri, C. V. Jawahar

Abstract

For individuals with Amyotrophic Lateral Sclerosis (ALS), the loss of verbal communication is a significant challenge. While many patients retain the ability to mouth words, their movements are often irregular and difficult for standard AI models to interpret. This paper presents a personalized lipreading system that utilizes a "one-shot" learning approach, requiring only a single real-world video example per word. By leveraging synthetic data augmentation and a novel Variational Encoder architecture for domain adaptation, the system achieves 83.2% top-5 accuracy for an ALS patient, providing a viable communication tool with minimal data collection effort.

Introduction

Amyotrophic Lateral Sclerosis (ALS) leads to progressive muscle weakness, often resulting in the loss of speech. Although patients may still mouth words, these movements are frequently labored and inconsistent compared to healthy speakers. Current lipreading models, typically trained on large datasets like "Lipreading in the Wild" (LRW), struggle to generalize to the unique articulatory patterns of medical patients. Furthermore, the physical strain of recording extensive training data makes traditional deep learning approaches impractical for this population. This research addresses the need for a highly accurate, personalized lipreading model that can be trained using extremely limited real-world data.

Methodology

The proposed system employs a three-stage pipeline designed to bridge the gap between limited real-world data and the requirements of deep learning models.

Overview of the personalized lipreading pipeline, from one-shot training to real-time inference.

1. Synthetic Data Augmentation

To overcome the "one-shot" limitation, the researchers "hallucinate" a larger training set. Using Text-to-Speech (TTS) models (FastSpeech2 and GlowTTS), they generate diverse audio clips for target words. These are then processed by Wav2Lip, which adapts a generic video of the patient to match the synthetic audio, creating a large volume of synthetic training data.

2. Variational Encoders (VE)

The core innovation is a Variational Encoder that focuses on task-relevant latent distributions. Unlike standard VAEs that prioritize input reconstruction, this VE is driven by downstream classification loss. This allows the model to learn the essential features of lip movement necessary for word recognition without the computational overhead of full video reconstruction.

3. Domain Adaptation

The VE facilitates domain adaptation by treating synthetic videos as the source domain and the single real example as the target. By minimizing the distance between these distributions in latent space, the model learns to recognize the patient's actual speaking style, effectively generalizing from "fake" training data to real-world performance.

Datasets and Experimental Setup

The study evaluated the model across several distinct datasets to test its versatility and robustness.

Diverse set of speakers used in the study, including the ALS patient, hearing-impaired individuals, and a control speaker.
  • Personalized Medical Dataset: Included one ALS patient (200-word vocabulary) and three hearing-impaired speakers with irregular lip movements.
  • Joe Biden Dataset: Served as a control to evaluate adaptation to specific accents and high-quality video outside the standard LRW distribution.
  • LRW (Lipreading in the Wild): Used for initial pre-training of the backbone architectures (ResNet18 + BiGRU/LTCN).

Key Results

The personalized approach demonstrated significant improvements over baseline models:

  • ALS Patient Accuracy: The baseline model (one-shot real data only) achieved 49.3% top-1 accuracy. The full system reached 68.1% top-1 and 83.2% top-5 accuracy.
  • Cross-Disability Performance: The model achieved 73.5% top-5 accuracy for speakers with hearing impairments.
  • Two-Shot Scaling: Providing just one additional real example (two-shot learning) increased the ALS patient's top-5 accuracy to nearly 90%.

Discussion and Conclusion

The success of this system lies in its ability to "hallucinate" variations of a single real example in latent space, making it robust to the natural inconsistencies of an ALS patient's movements. By removing the decoder and focusing on classification-relevant features, the architecture is both efficient and effective.

However, limitations remain. The system currently operates on a closed vocabulary of 200 words and requires external assistance for initial setup and label verification. Future work will focus on expanding to open-vocabulary recognition and further automating the domain adaptation process to enhance independence for users with severe motor disabilities.

Summary updated

BibTeX
@inproceedings{sen2021personalized,
  author = {Bipasha Sen and Aditya Agarwal and Rudrabha Mukhopadhyay and Vinay P. Namboodiri and C. V. Jawahar},
  title = {{Personalized One-Shot Lipreading for an ALS Patient}},
  booktitle = {BMVC},
  year = {2021},
}
Figure