Towards MOOCs for Lip Reading: Using Synthetic Talking Heads to Train Humans in Lipreading at Scale

Aditya Agarwal, Bipasha Sen, Rudrabha Mukhopadhyay, Vinay Namboodiri, C. V. Jawahar

Abstract

For nearly 430 million people worldwide with hearing loss, lipreading is a primary mode of communication. However, learning this skill is hindered by a lack of accessible, diverse, and affordable training resources. Traditional methods for creating lipreading content are labor-intensive, requiring months of manual recording with hired actors, which results in a "diversity bottleneck" regarding languages, accents, and speakers. This paper investigates the use of AI-generated synthetic talking heads to automate the creation of lipreading training materials. We propose an end-to-end pipeline that leverages state-of-the-art generative models and text-to-speech (TTS) technology. Our human evaluation demonstrates that synthetic videos are as effective as real human footage for training, and that localized accents significantly improve performance, paving the way for large-scale, localized lipreading MOOCs.

The Diversity Bottleneck in Lipreading Training

Existing online lipreading resources suffer from significant limitations. Because they rely on manual production cycles involving professional actors and studio environments, they are expensive and slow to produce. This leads to platforms that are limited to a few languages (primarily English), a handful of faces, and specific accents. For a global population with diverse linguistic needs, this lack of localized content is a major barrier to skill development.

The limitations of the current manual pipeline for creating lipreading training platforms.

An Automated Pipeline for Synthetic Training Content

To overcome these hurdles, we developed a fully automated pipeline capable of generating unlimited training content in any language or accent in less than a day. The system consists of four primary modules:

  1. Video Selection: The system scrapes "in-the-wild" talking head videos from platforms like YouTube to ensure a wide variety of identities, backgrounds, and head poses.
  2. Audio Selection & TTS: Vocabulary is curated from millions of online articles and converted into synthetic speech using models like FastSpeech2, allowing for the generation of specific regional accents.
  3. Audio-Visual Alignment: To ensure the training is effective, the system precisely aligns the synthetic audio with the video, removing any residual lip movements from the original source that might confuse the trainee.
  4. Wav2Lip Generation: A generative model modifies the lip movements of the source video to match the synthetic audio perfectly while preserving the original speaker's identity and environment.
The proposed automated approach for large-scale lipreading training video generation.

Key Results and Insights

Our study, involving 50 hearing-impaired participants, yielded several critical findings:

  • Performance Parity: Using Bayesian Equivalence Analysis (BEST), we found that humans perform just as well when learning from AI-generated lips as they do from real human lips. In Word-Level (WL) tasks, the mean scores for Real American English and Synthetic American English were nearly identical, suggesting no "uncanny valley" effect that would hinder learning.
  • The Importance of Localized Accents: The study highlighted a significant "accent gap." Indian participants performed substantially better when the synthetic video used an Indian accent compared to an American one. This confirms that lipreading is not just about visual shapes but is deeply tied to the viewer's native linguistic and phonetic expectations.
  • Efficiency and Scalability: While traditional pipelines take months to produce a single course, our automated approach can generate a massive, diverse training library in less than 24 hours with no manual effort.

Conclusion and Future Work

This research demonstrates that synthetic talking heads are a viable and effective tool for training humans in lipreading. By automating the production process, we can finally create diverse, localized, and free MOOCs that cater to the specific needs of the global hearing-impaired community. Future work will focus on expanding the pipeline to tonal and non-English languages and integrating these tools into interactive, adaptive learning platforms.

Summary updated

Abstract

Many people with some form of hearing loss consider lipreading as their primary mode of day-to-day communication. However, finding resources to learn or improve one's lipreading skills can be challenging. This is further exacerbated in COVID$19$ pandemic due to restrictions on direct interactions with peers and speech therapists. Today, online MOOCs platforms like Coursera and Udemy have become the most effective form of training for many kinds of skill development. However, online lipreading resources are scarce as creating such resources is an extensive process needing months of manual effort to record hired actors. Because of the manual pipeline, such platforms are also limited in the vocabulary, supported languages, accents, and speakers, and have a high usage cost. In this work, we investigate the possibility of replacing real human talking videos with synthetically generated videos. Synthetic data can be used to easily incorporate larger vocabularies, variations in accent, and even local languages, and many speakers. We propose an end-to-end automated pipeline to develop such a platform using state-of-the-art talking heading video generator networks, text-to-speech models, and computer vision techniques. We then perform an extensive human evaluation using carefully thought out lipreading exercises to validate the quality of our designed platform against the existing lipreading platforms. Our studies concretely point towards the potential of our approach for the development of a large-scale lipreading MOOCs platform that can impact millions of people with hearing loss.
BibTeX
@inproceedings{agarwal2023towards,
  author = {Aditya Agarwal and Bipasha Sen and Rudrabha Mukhopadhyay and Vinay Namboodiri and C. V. Jawahar},
  title = {{Towards MOOCs for Lip Reading: Using Synthetic Talking Heads to Train Humans in Lipreading at Scale}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2023},
  doi = {10.1109/wacv56688.2023.00225},
}

Loading related content...

Figure