Speech-Driven Facial Animations Improve Speech-in-Noise Comprehension of Humans

Enrico Varano, Konstantinos Vougioukas, Pingchuan Ma, Stavros Petridis, Maja Pantić, Tobias Reichenbach

Abstract

Understanding speech in noisy environments is a significant challenge, particularly for individuals with hearing impairments. While visual cues from a speaker's face can substantially improve comprehension—a phenomenon known as the audiovisual benefit—such visual information is not always available. This paper explores the use of Generative Adversarial Networks (GANs) to synthesize photorealistic talking faces from a single still image and a speech recording. We demonstrate that these end-to-end synthesized videos are indistinguishable from natural videos to human observers and, crucially, provide a significant boost to speech-in-noise (SiN) comprehension for both humans and automated speech recognition systems.

Introduction

In difficult listening environments, such as crowded social gatherings, the brain relies heavily on visual information to disambiguate acoustic signals. This "visual hearing aid" is vital for effective communication. However, in many scenarios—telecommunications, radio broadcasts, or when speakers are wearing masks—this visual stream is missing. Recent advances in deep learning suggest that we can "hallucinate" this missing information. This study investigates whether AI-generated facial animations can provide the same linguistic utility as natural video to help a listener understand speech-in-noise (SiN).

Methodology

The researchers employed a Generative Adversarial Network (GAN) architecture (specifically the model by Vougioukas et al., 2020) to produce end-to-end synthesized videos. Unlike traditional methods that rely on intermediate phonetic representations or facial landmarks, this model generates pixels directly from the raw audio waveform and a single identity image.

The Face Animator architecture synthesizes a sequence of video frames from a single still image and an audio waveform.

To evaluate the effectiveness of these animations, the team conducted three distinct tests:

  1. Turing Realism Test: A large-scale study to see if humans could distinguish between real and GAN-generated videos.
  2. Human Comprehension Study: Participants listened to noisy sentences under three conditions: Audio-only, Synthetic Audiovisual (AV), and Natural AV.
  3. Machine Comprehension: Testing an Audiovisual Automated Speech Recognizer (AVSR) to see if synthetic video improved machine "hearing."

Key Results

The results confirmed that the GAN-generated faces are highly realistic; participants in the Turing test performed at a 50% chance level, indicating they could not distinguish between real and synthetic videos.

A sequence of synthesized facial frames demonstrating the photorealistic lip movements generated by the model.

More importantly, the synthetic video provided a measurable benefit to comprehension. In noisy conditions, human comprehension improved from a ~51% baseline (audio-only) to ~62% with synthetic AV—a 22% relative improvement. While natural video still provided the highest benefit (~71% comprehension), the boost from the AI was significant. Machine performance also improved, with the AVSR showing a 13% increase in accuracy when provided with the synthetic visual stream.

Discussion and Insights

A striking finding was that "realism" does not necessarily equal "information." Even though humans could not tell the fake videos from the real ones, natural videos were still more helpful for comprehension. This suggests that while the GAN captures the look of speech perfectly, it may slightly smooth over the fine-grained kinematics that the brain uses for deep decoding.

Confusion matrices revealed that the GAN was particularly effective at helping humans distinguish the letter "b" from other consonants, though it struggled with "n" and "m" sounds, which involve subtle internal mouth movements harder to learn from external video pixels alone.

Limitations and Future Work

The study utilized the GRID corpus, which consists of highly structured sentences. Future research must address whether these models generalize to spontaneous, emotional, or complex real-world vocabulary. Furthermore, real-time applications would require near-zero latency; any lag between audio and synthesized video could hinder rather than help comprehension. Future work will likely focus on "noise-hardened" synthesizers capable of generating faces even when the input audio is already corrupted by heavy background noise.

Summary updated

Abstract

Understanding speech becomes a demanding task when the environment is noisy. Comprehension of speech in noise can be substantially improved by looking at the speaker's face, and this audiovisual benefit is even more pronounced in people with hearing impairment. Recent advances in AI have allowed to synthesize photorealistic talking faces from a speech recording and a still image of a person's face in an end-to-end manner. However, it has remained unknown whether such facial animations improve speech-in-noise comprehension. Here we consider facial animations produced by a recently introduced generative adversarial network (GAN), and show that humans cannot distinguish between the synthesized and the natural videos. Importantly, we then show that the end-to-end synthesized videos significantly aid humans in understanding speech in noise, although the natural facial motions yield a yet higher audiovisual benefit. We further find that an audiovisual speech recognizer (AVSR) benefits from the synthesized facial animations as well. Our results suggest that synthesizing facial motions from speech can be used to aid speech comprehension in difficult listening environments.
BibTeX
@inproceedings{varano2022speech,
  author = {Enrico Varano and Konstantinos Vougioukas and Pingchuan Ma and Stavros Petridis and Maja Pantić and Tobias Reichenbach},
  title = {{Speech-Driven Facial Animations Improve Speech-in-Noise Comprehension of Humans}},
  booktitle = {Frontiers in Neuroscience},
  year = {2022},
  doi = {10.3389/fnins.2021.781196},
}
Figure