Towards MOOCs for Lipreading: Using Synthetic Talking Heads to Train Humans in Lipreading at Scale

Aditya Agarwal, Bipasha Sen, Rudrabha Mukhopadhyay, Vinay P. Namboodiri, C. V. Jawahar

Abstract

Lipreading is a vital communication tool for millions with hearing loss, yet existing training resources are limited by high production costs and a lack of linguistic diversity. This paper introduces a fully automated pipeline for generating large-scale lipreading training platforms (MOOCs) using AI-generated synthetic talking heads. By integrating text-to-speech models with lip-syncing networks like Wav2Lip, the system produces unlimited training videos across various languages and accents. A user study with 50 hearing-impaired participants confirms that synthetic characters are as effective for training as real human actors, paving the way for accessible, global lipreading education.

Introduction

For individuals with hearing impairment, lipreading serves as a primary or supplementary mode of communication. However, the current landscape of lipreading education is dominated by "boutique" operations that suffer from limited scale, high production costs, and a lack of regional diversity. Traditional methods require hiring actors and manual studio recording, a process that can take months to produce a single course.

The proposed automated pipeline significantly reduces production time and costs compared to traditional manual methods.

The central challenge addressed here is whether synthetic talking heads can replace real human videos without compromising pedagogical effectiveness.

Methodology

The researchers developed an end-to-end automated pipeline to generate a "Synthetic Talking Head Database." The process begins with scraping "in-the-wild" face videos from YouTube to ensure diversity in identity and environment. Simultaneously, a massive vocabulary is curated from online resources.

Key technical components include:

  • Text-to-Speech (TTS): Utilizing models like FastSpeech2 to generate audio in diverse accents (e.g., American and Indian English).
  • Audio-Video Alignment: A custom module ensures synthetic audio is precisely aligned with the visual mouth regions of the source video, minimizing artifacts by adding silence where speech is not intended.
  • Wav2Lip Generation: This state-of-the-art model modifies the source video's lip movements to match the synthetic audio while preserving the original speaker's identity, background, and head pose.
Comparison of real versus synthetic video frames, demonstrating the high fidelity of the generated lip movements.

Results and Key Insights

A rigorous evaluation involving 4,200 synthetic videos and 50 hearing-impaired participants yielded several critical findings:

  1. Real vs. Synthetic Parity: Using Bayesian Equivalence Analysis (BEST), the authors proved that user performance on synthetic videos was statistically equivalent to performance on real human videos. This suggests the "uncanny valley" is not a barrier to learning.
  2. The Importance of Accents: Participants performed significantly better when training with their native accent (e.g., Indian English) compared to a foreign one (American English). This underscores the necessity of the pipeline's ability to generate localized content quickly.
  3. Contextual Advantage: Users achieved higher scores when lipreading full sentences with contextual clues (e.g., "in a restaurant") compared to isolated words.

Conclusion

This research demonstrates that synthetic talking heads are a viable, scalable substitute for real actors in lipreading education. By reducing production time from months to less than a day, this automated pipeline enables the creation of a "Duolingo for Lipreading"—a free, diverse, and globally accessible training platform for the hearing-impaired community.

Summary updated

Abstract

Many people with some form of hearing loss consider lipreading as their primary mode of day-to-day communication. However, finding resources to learn or improve one’s lipreading skills can be challenging. This is further exacerbated in the COVID-19 pandemic due to restrictions on direct interactions with peers and speech therapists. Today, online MOOCs platforms like Coursera and Udemy have become the most effective form of training for many types of skill development. However, online lipreading resources are scarce as creating such resources is an extensive process needing months of manual effort to record hired actors. Because of the manual pipeline, such platforms are also limited in vocabulary, supported languages, accents, and speakers and have a high usage cost. In this work, we investigate the possibility of replacing real human talking videos with synthetically generated videos. Synthetic data can easily incorporate larger vocabularies, variations in accent, and even local languages and many speakers. We propose an end-to-end automated pipeline to develop such a platform using state-of-the-art talking head video generator networks, text-to-speech models, and computer vision techniques. We then perform an extensive human evaluation using carefully thought-out lipreading exercises to validate the quality of our designed platform against the existing lipreading platforms. Our studies concretely point toward the potential of our approach in developing a large-scale lipreading MOOC platform that can impact millions of people with hearing loss.
BibTeX
@inproceedings{agarwal2023towards,
  author = {Aditya Agarwal and Bipasha Sen and Rudrabha Mukhopadhyay and Vinay P. Namboodiri and C. V. Jawahar},
  title = {{Towards MOOCs for Lipreading: Using Synthetic Talking Heads to Train Humans in Lipreading at Scale}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2023},
  doi = {10.1109/wacv56688.2023.00225},
}
Figure