Facial Landmark Predictions with Applications to Metaverse

Qiao Han, Jun Zhao, Kwok-Yan Lam

Abstract

This research aims to make metaverse characters more realistic by adding lip animations learned from videos in the wild. To achieve this, our approach extends the Tacotron 2 text-to-speech synthesizer to generate lip movements together with a mel spectrogram in one pass. The encoder and gate layer weights are pre-trained on the LJ Speech 1.1 dataset, while the decoder is retrained on 93 clips of TED talk videos extracted from the LRS 3 dataset. Our novel decoder predicts displacement in 20 lip landmark positions across time, using labels automatically extracted by the OpenFace 2.0 landmark predictor. Training converged in 7 hours using less than 5 minutes of video. We conducted an ablation study for Pre/Post-Net and pre-trained encoder weights to demonstrate the effectiveness of transfer learning between audio and visual speech data.

Introduction

In the evolving landscape of the Metaverse, the realism of digital avatars is paramount for immersive communication. Non-verbal cues, particularly lip movements, are essential for making digital characters feel "alive" and ensuring speech is easily understood. However, current lip-syncing methods face significant hurdles. Most existing systems are "audio-driven," meaning they generate movement by analyzing a sound signal. This introduces inherent latency as the system waits for audio processing. Furthermore, creating realistic animations typically requires massive, hand-labeled datasets or expensive 3D face scans that are difficult to scale.

This paper proposes a text-driven approach that predicts 3D lip movements directly from text embeddings. By bypassing audio delays and simplifying the animation pipeline, this method enables high-quality, real-time applications such as NPC dialogue and player communication in virtual environments.

Methodology

Architecture Adaptation

The proposed model extends the Tacotron 2 architecture, a standard in neural speech synthesis. While the original Tacotron 2 is designed to output mel-spectrograms for audio, this modified version is optimized to output the 3D coordinates of 20 specific facial landmarks surrounding the lips.

Through rigorous ablation studies, the researchers identified that simplifying the architecture yielded better results for visual data. Specifically, removing the standard "Pre-Net" and "Post-Net" components—which are essential for audio fidelity—actually stabilized facial landmark predictions and accelerated model convergence.

Displacement Mapping

A key innovation in this approach is the use of displacement mapping. Rather than forcing the model to learn the absolute 3D coordinates of a mouth, it is trained to predict the displacement (movement) from a neutral reference frame. This technique ensures the model remains invariant to the speaker's specific head position or face shape, allowing for better generalization across different avatar types.

Transfer Learning

The model leverages transfer learning by utilizing an encoder pre-trained on the LJ Speech 1.1 dataset (containing 24.6 hours of audio). This allows the model to utilize existing knowledge of linguistic structures to understand how specific phonemes translate into visual movements. This cross-modal transfer between audio and visual speech data significantly reduces the amount of video data required for training.

Experimental Setup and Datasets

The study utilized a combination of audio and video datasets:

  • LJ Speech 1.1: Used for initial encoder pre-training to establish text embedding proficiency.
  • LRS 3 (Lip Reading Sentences 3): A subset of 93 TED talk clips (totaling less than 5 minutes of video) was used to fine-tune the landmark decoder.
  • OpenFace 2.0: This tool was employed to automatically generate "ground truth" 68-point facial landmark labels from the raw video, removing the need for manual annotation.

Results and Discussion

Efficiency and Accuracy

The model achieved an average error of only 8mm from the ground truth 3D locations of the speaker’s lips. Remarkably, the entire training process was completed in just 7 hours on a single Nvidia A100 GPU. Due to the pre-trained encoder, the model converged in fewer than 5,000 iterations, whereas the original Tacotron 2 often requires 50,000 iterations for high-quality audio synthesis.

Qualitative Observations

Visual comparisons demonstrate that the text-driven approach produces natural-looking lip closures for difficult consonants (e.g., "M" or "B" sounds). These results were found to be comparable to, and in some cases more distinct than, traditional noise-resilient audio-driven models.

The "Less is More" Principle

The researchers compared a model predicting all 68 facial landmarks (Model A) against a focused model predicting only the 20 lip landmarks (Model B). Model B performed significantly better. By focusing the loss function specifically on the lips, the researchers reduced model complexity and eliminated "noisy" movements in the eyes or jaw that did not correlate directly with the speech input.

Conclusion and Future Work

This research demonstrates that high-quality lip animation can be achieved with minimal training data by leveraging transfer learning and architectural simplifications. While the current model excels at technical lip-syncing, it does not yet account for emotional context or speaker tone.

Future research will focus on:

  1. Emotional Embeddings: Incorporating context to allow for expressive animations (e.g., whispering vs. shouting).
  2. Scaling: Exploring self-supervised methods, such as masked autoencoders, to utilize thousands of hours of unlabeled video.
  3. Edge Optimization: Refining the model for standalone VR hardware where power consumption and inference speed are critical constraints.

Summary updated

Abstract

This research aims to make metaverse characters more realistic by adding lip animations learnt from videos in the wild. To achieve this, our approach is to extend Tacotron 2 text-to-speech synthesizer to generate lip movements together with mel spectrogram in one pass. The encoder and gate layer weights are pre-trained on LJ Speech 1.1 data set while the decoder is retrained on 93 clips of TED talk videos extracted from LRS 3 data set. Our novel decoder predicts displacement in 20 lip landmark positions across time, using labels automatically extracted by OpenFace 2.0 landmark predictor. Training converged in 7 hours using less than 5 minutes of video. We conducted ablation study for Pre/Post-Net and pre-trained encoder weights to demonstrate the effectiveness of transfer learning between audio and visual speech data.
BibTeX
@inproceedings{han2022facial,
  author = {Qiao Han and Jun Zhao and Kwok-Yan Lam},
  title = {{Facial Landmark Predictions with Applications to Metaverse}},
  booktitle = {WF-IoT},
  year = {2022},
  url = {https://openreview.net/forum?id=Exh4iodKB0RA},
}
Figure