Active appearance models for photorealistic visual speech synthesis

Wesley Mattheyses, Lukas Latacz, Werner VerhelstInterspeech 2010

TL;DR

This paper introduces a novel approach to creating more realistic synthetic talking faces by using Active Appearance Models (AAMs) to disentangle the shape and texture of facial movements. This allows for better control over the smoothness of lip movements and the overall appearance, leading to significantly improved perceived visual speech quality.

What Problem Does This Paper Address?

Generating realistic visual speech for synthetic talking heads is a significant challenge in audiovisual speech synthesis. Traditional methods often struggle to balance the need for smooth, natural-looking facial movements with the requirement for clear articulation. When concatenating pre-recorded speech segments, abrupt changes in lip shape or texture can easily break the illusion of naturalness, making the synthesized speech appear "mumbled" or artificial. This paper addresses the gap in controlling these visual aspects independently to achieve a more convincing and natural visual output.

Methods

The core of this paper lies in the application and adaptation of Active Appearance Models (AAMs) for visual speech synthesis. AAMs are statistical models that represent images by decomposing them into two key components: shape (defined by landmark points indicating facial features like lips) and texture (the pixel information within those shapes).

Here's a breakdown of their approach:

  • AAM Training: They train an AAM specifically on the mouth area of talking faces. This focuses the model on speech-relevant movements and appearances. The training involves an iterative process of manual landmarking, initial AAM training, automatic tracking of a larger dataset, and then refinement based on clustering and manual correction. This ensures a high-quality AAM that captures significant variations in the training data.
  • Visual Speech Synthesis: The trained AAM transforms video frames into shape and texture parameters. When synthesizing speech, segments are selected from a database, and their corresponding AAM parameter trajectories are concatenated.
  • Controlled Smoothing: A key innovation is the ability to independently smooth the shape and texture trajectories during concatenation. This allows for smoothing the overall appearance (texture) to ensure seamless transitions while maintaining sharper lip movements (shape) to avoid under-articulation.
  • Database Normalization: To further improve synthesis quality, they propose a novel database normalization strategy. This strategy identifies and sets to zero model parameters that are not correlated with speech (e.g., slight head rotations, illumination changes). This is achieved by comparing intra-phoneme variations with overall parameter variations, and by analyzing the difference between synthesized and original speech trajectories.

Datasets

  • LIPS2008 audiovisual speech database: This database was used for training the AAM and for preparing the visual speech database for synthesis.
  • Custom synthesized sentences: Used for evaluating the database normalization strategy. These sentences were synthesized from the database and then compared to their original counterparts.

Key Insights and Ideas

  • Shape vs. Texture Control: The ability of AAMs to disentangle shape and texture is crucial. This allows for a fine-grained control over visual speech synthesis, enabling separate smoothing of appearance and articulation.
  • Normalization for Realism: Non-speech-related variations in a database (like lighting or head pose) can severely degrade synthesis quality. Normalizing these parameters to zero significantly improves the perceived naturalness.
  • Iterative AAM Training: A multi-stage iterative approach to AAM training, combining manual and automatic steps, is effective for building high-quality models that preserve image detail.
  • Perception Matters: The ultimate measure of success is human perception. The paper emphasizes the importance of user studies to validate the effectiveness of their techniques.

Key Results

  • Improved Synthesis Quality: A perception experiment showed that using the normalized database significantly improved the perceived quality of synthetic visual speech. A Wilcoxon signed rank test yielded a Z-value of -8.265 (p < 0.01), indicating a highly significant improvement.
  • Effective Normalization: The proposed normalization strategy successfully reduced variations not correlated with speech, leading to smoother and more natural mouth movements.
  • AAMs as a Powerful Tool: The paper demonstrates that AAMs are well-suited for visual speech synthesis, offering a way to manage the complexities of facial animation.

Limitations and Future Work

  • Manual Labor: While the AAM training is iterative, it still involves manual landmarking, which can be time-consuming.
  • AAM Limitations: The paper acknowledges that AAMs model specific aspects of an image and might not capture all nuances of facial expression.
  • Future Work: The authors suggest further exploration of how to exploit the AAM parameters for more sophisticated synthesis, potentially leading to even more natural and expressive talking heads. They also point to a demo website for sample syntheses.

Summary updated

BibTeX
@inproceedings{mattheyses2010active,
  author = {Wesley Mattheyses and Lukas Latacz and Werner Verhelst},
  title = {{Active appearance models for photorealistic visual speech synthesis}},
  booktitle = {Interspeech},
  year = {2010},
}

Loading related content...

Figure