TL;DR
This paper presents an enhanced Czech talking head system that synthesizes realistic visual speech animation from text. The system leverages a novel animation schema for more precise lip and tongue movements and a data-driven synthesis method to capture complex coarticulation effects, aiming to improve lipreading support and human-computer interaction.
What Problem Does This Paper Address?
The creation of realistic talking head systems, which animate a virtual character speaking, is crucial for improving human-computer interaction, particularly for applications like lipreading training for the hearing impaired. While text-to-speech (TTS) systems can generate audio, synchronizing this with natural-looking visual speech, especially for languages like Czech with its specific phonetic nuances, presents a significant challenge. Existing methods often struggle with precise articulation, particularly in the lip and tongue areas, and accurately modeling the subtle coarticulation effects (how sounds influence each other) that are vital for natural speech. This paper addresses the need for a more sophisticated Czech talking head system that can produce visually accurate and synchronized speech animation, overcoming limitations in existing animation schemas and synthesis techniques.
Methods
The core of the system is a 3D animation schema built upon polygonal meshes representing facial features. This schema uses feature points and deformation zones to control the animation. A key innovation is the use of 3D cubic spline curves defined by control points to model deformations, offering more flexibility and precision, especially around the lips. The system handles overlapping deformation zones by introducing a weighted sum of influences from multiple curves, preventing unnatural displacements.
For visual speech parameterization, the paper adopts the MPEG-4 standard and employs Principal Component Analysis (PCA) to reduce the dimensionality of feature point data, resulting in parameters like lip opening, protrusion, and jaw rotation.
The synthesis method is data-driven, specifically using a selection of articulatory targets approach. This method employs Classification and Regression Trees (CART) to predict articulatory targets based on phonetic context. The system formulates regression questions to capture labial coarticulation effects, considering factors like phonetic context, phoneme type (vowel, stop, fricative), and acoustic properties. The output is a set of articulatory trajectories, which are then interpolated to create the animation frames.
Datasets
The paper describes the acquisition of two audiovisual corpora for Czech speech:
- Corpus 1: Collected using optical tracing with retro-reflex marks on speakers' faces. It comprises 318 sentences and includes artificially constructed CVC and VCV words.
- Corpus 2: Computed articulatory trajectories using template matching on video records. This corpus consists of 974 sentences and features a speech therapy expert as the speaker.
Both corpora are time-labeled to phoneme speech segments using HMMs and the Viterbi algorithm. These datasets are used for training the synthesis method and evaluating the system.
Key Insights and Ideas
- 3D Spline Curves for Precision: Using 3D cubic spline curves for deformation modeling, rather than simpler methods, allows for a more accurate representation of facial movements, particularly for the complex shapes of the lips and tongue.
- Handling Overlapping Deformations: The proposed method for managing overlapping influence zones in the animation schema is crucial for achieving smooth and natural transitions, avoiding the "jagged" or unnatural movements that can occur when multiple deformation areas intersect.
- Data-Driven Coarticulation: The CART-based synthesis method, by explicitly modeling coarticulation effects through carefully designed regression questions, allows the system to capture the subtle ways sounds influence each other in fluent speech, leading to more realistic visual output.
- MPEG-4 Parameterization: Adopting a standard like MPEG-4 for visual speech parameterization provides a structured and potentially interoperable way to represent facial movements.
Key Results
The paper reports that the Czech talking head system has been evaluated and shows significant benefits in the intelligibility of synthesized visual speech, suggesting it can be a valuable tool for lipreading support. While specific quantitative results (e.g., percentage improvement in intelligibility) are not detailed in this excerpt, the qualitative findings highlight the system's ability to produce more precise approximations of lip and tongue movements and to capture labial coarticulation effects. The system is also being applied to the synthesis of Czech signed speech.
Limitations and Future Work
- Limitations: The paper acknowledges that the animation schema, while improved, still relies on polygonal meshes and feature points, which inherently have limitations in capturing the full complexity of human facial dynamics. The reliance on specific datasets and synthesis methods might also limit generalizability.
- Future Work: The authors mention plans to conduct perception studies with deaf children to further enhance the system with new features. The system is also being extended for the synthesis of Czech signed speech, indicating a broader application scope. Further research into more sophisticated animation models and synthesis techniques is implied.
Summary updated