Modeling Labial Coarticulation with Bidirectional Gated Recurrent Networks and Transfer Learning

Théo Biasutto--Lervat, Sara Dahmani, Slim OuniInterspeech 2019

TL;DR

This paper tackles the challenge of generating realistic lip movements for virtual avatars synchronized with speech. They propose a novel approach using bidirectional gated recurrent networks (Bi-GRUs) combined with transfer learning techniques to effectively model the complex phenomenon of labial coarticulation, achieving significant improvements in accuracy and training efficiency.

What Problem Does This Paper Address?

Creating believable virtual characters that speak requires synchronizing lip movements with audio. This is complicated by "coarticulation," where the pronunciation of one sound influences the sounds around it, leading to subtle but crucial variations in lip shape. Existing methods often struggle with this, especially when dealing with limited audiovisual data. Acquiring large, synchronized audiovisual corpora is expensive and time-consuming, posing a significant bottleneck for deep learning approaches. This paper aims to overcome this data scarcity by developing a robust model that can learn coarticulation effectively and efficiently.

Methods

The core of the proposed solution lies in a Bidirectional Gated Recurrent Network (Bi-GRU) architecture. Bi-GRUs are chosen because they can process sequential data in both forward and backward directions, allowing them to capture temporal dependencies crucial for understanding how past and future speech sounds influence current lip movements.

To address the data scarcity, the authors employ transfer learning through a pretraining strategy. Instead of randomly initializing the network, they leverage dimensionality reduction techniques to inject prior knowledge into the model. Specifically, they explore three methods:

  • Principal Component Analysis (PCA): Uses eigenvectors to create a compressed representation of facial movements.
  • Autoencoders: Learn a compressed latent representation of facial data and a decoder to reconstruct it.
  • Blendshape Decomposition: A hand-crafted method from the animation community that uses a set of predefined facial shapes (keyshapes) to model deformations.

The idea is to initialize the network's layers with the learned representations from these techniques, effectively giving the model a head start in understanding facial articulation before it even begins training on the specific audiovisual task.

Datasets

The authors used a custom audiovisual corpus consisting of approximately 4 hours of French speech. This corpus was recorded using a motion-capture system with 63 sensors placed on a professional actress's face, though only 44 sensors corresponding to the lower face were used for this study.

Key Insights and Ideas

  • Coarticulation is Key: The paper strongly emphasizes that accurately modeling labial coarticulation is essential for generating natural-looking lip movements.
  • Bi-GRUs are Well-Suited: Bidirectional GRUs prove effective in capturing the complex temporal dynamics of speech-driven facial articulation.
  • Transfer Learning is a Game-Changer for Data Scarcity: Injecting pre-computed latent representations significantly improves training efficiency and model performance, especially when dealing with limited datasets.
  • Data-Driven vs. Hand-Crafted Initialization: Surprisingly, hand-crafted blendshape decomposition performed comparably to data-driven PCA and autoencoders, suggesting that well-designed prior knowledge can be as effective as learned representations.
  • Layer Normalization Helps: Even without parameter injection, using layer normalization in the network architecture demonstrably improves training.

Key Results

  • Improved Accuracy: The pretraining strategy, regardless of the method (PCA, Blendshape, Autoencoder), led to significant improvements in performance. The average RMSE (Root Mean Squared Error) achieved was around 1.3mm.
  • Faster Training: The number of epochs required to reach optimal performance was drastically reduced. For example, the baseline model needed about 70 epochs, while the pretrained models (PCA, Blendshape, Autoencoder) achieved similar results in about 10 epochs, representing a ~7x speedup.
  • Reduced Variability: Pretraining also led to more consistent final performances, with less variability across multiple training runs.
  • Comparison of Methods: PCA and Blendshape initialization showed slightly better overall performance compared to the Autoencoder, though all pretrained models significantly outperformed the randomly initialized baseline.

Limitations and Future Work

  • Limited Dataset Size: While the paper addresses data scarcity, 4 hours is still relatively small for deep learning. The authors acknowledge that further improvements could be made with larger datasets.
  • Focus on Lower Face: The study focused on the lower part of the face, excluding eyelids and eyebrows. Future work could incorporate these elements for more complete facial animation.
  • Phonetic Sequence Input: The model relies on phonetic sequences as input. Future work aims to explore injecting knowledge directly from the phonetic sequences themselves, inspired by advancements in Natural Language Processing (NLP) like ELMo and BERT, to potentially leverage even larger amounts of readily available phonetic data.
  • Perceptual Study: The authors plan to conduct a perceptual study to assess the subjective quality of the generated lip movements.

Summary updated

BibTeX
@inproceedings{biasuttolervat2019modeling,
  author = {Théo Biasutto--Lervat and Sara Dahmani and Slim Ouni},
  title = {{Modeling Labial Coarticulation with Bidirectional Gated Recurrent Networks and Transfer Learning}},
  booktitle = {Interspeech},
  year = {2019},
}
Figure