Evaluation of a formant-based speech-driven lip motion generation

Carlos T. Ishi, Chaoran Liu, Hiroshi Ishiguro, Norihiro HagitaInterspeech 2012

TL;DR

This paper presents an improved method for generating realistic lip movements for humanoid robots and animated characters directly from speech. By analyzing vowel formants, the system can control lip height and, for the first time, partially control lip width, leading to more natural-looking lip synchronization than previous audio-only methods.

What Problem Does This Paper Address?

Creating realistic lip synchronization for telepresence robots and virtual avatars is a significant challenge. Traditional methods often rely on expensive motion capture or vision-based tracking, which can be sensitive to lighting and speaker variations. The authors aim to develop a robust, real-time system that generates lip motion directly from an operator's voice, enabling more natural and cost-effective telepresence. Specifically, they address the need for accurate lip height and width control, which are crucial for conveying emotion and speech naturally.

Methods

The core of the proposed method is a formant-based approach to lip motion generation. Here's a breakdown of the key steps:

  • Formant Extraction: The system analyzes the input speech signal to extract the first two formants (F1 and F2), which are acoustic resonances related to vocal tract shape. This is done using Linear Predictive Coding (LPC) and peak picking on the smoothed spectrum.
  • Speaker Normalization: To account for variations in speakers' vocal tracts (due to gender, age, etc.), the formant space is normalized. This involves translating the origin of the formant space to a speaker-specific "vowel center" (representing the schwa vowel) and then scaling the coordinates. A novel aspect here is the automatic estimation of the LPC order based on the speaker's normalized vowel space, improving the correspondence between formants and spectral peaks.
  • Vowel Space Rotation: The normalized formant space is rotated counter-clockwise by approximately 25 degrees. This rotation aligns one of the new axes (logF1') with lip height and the other (logF2') with lip width.
  • Lip Shape Estimation:
    • Lip Height: Estimated directly from the rotated F1' (logF1') values. Higher logF1' values correspond to greater lip opening.
  • Lip Width: Estimated from the F2 values before rotation. Positive delta_lip_width values indicate lip spreading (like for /i/), while negative values indicate lip rounding (like for /o/).
  • Consonant Handling: For consonants, where formants are harder to estimate, the system imposes constraints on formant ranges and power values. A fixed lip height of 0.35 is used for consonants, and specific constraints are applied to detect fricative and affricative consonants.
  • Lip Width Control (Partial): A key innovation is the partial control of lip width. Since the android robot cannot round its lips, the system uses an actuator designed for stretching lip corners. This actuator is mapped to the estimated delta_lip_width values, allowing for a form of lip spreading.
  • Smoothing and Filtering: A moving average filter is applied to the generated lip height and width sequences to smooth out jerky movements.

Datasets

The paper describes experiments conducted using simultaneous recordings of audio, vision-based face parameter data, and motion-captured data from seven speakers (four male, three female) speaking in various languages (Japanese, English, Persian, Turkish). The authors do not introduce a new dataset but rather utilize existing recordings for evaluation.

Key Insights and Ideas

  • Formants as a Robust Speech-to-Lip Mapping: The paper strongly advocates for using vowel formants as a more direct and robust acoustic feature for lip motion generation compared to traditional methods like MFCCs, which require more complex models and have less straightforward phonetic interpretations.
  • Speaker Normalization is Crucial: The normalization step, which adjusts for individual speaker characteristics, is vital for achieving consistent and natural lip movements across different individuals. The automatic adjustment of LPC order based on this normalization is a clever improvement.
  • Partial Lip Width Control Enhances Naturalness: Even with the limitation of not being able to round lips, controlling lip corner stretching significantly improves the perceived naturalness of the lip motion on the android robot. This highlights the importance of even subtle lip movements.
  • Animated Lips Outperform Robots (Currently): The higher naturalness scores for animated lips suggest that while the formant-based method is effective, achieving truly human-like lip rounding and complex movements on a physical robot still presents challenges.

Key Results

  • Improved Naturalness: Subjective evaluations showed that the proposed formant-based method, especially with the partial lip width control, resulted in improved naturalness of lip motion compared to using only jaw control.
  • Animated Lips Achieved High Scores: Animated lips generated by the method achieved the highest naturalness scores (above 80 on a 0-100 scale), indicating the method's potential for creating highly realistic lip movements when full lip articulation is possible.
  • Partial Width Control Benefits: For the Geminoid-F robot, using both jaw and lip corner actuators ("jaw+lip corner") resulted in higher naturalness scores than using only the jaw actuator ("jaw only").
  • Language Independence: No significant differences in naturalness scores were found across different languages, suggesting the method's generalizability.

Limitations and Future Work

  • Formant Extraction Errors: The authors acknowledge that simple peak picking for formant extraction can fail, particularly for vowels like /o/ and /u/ where F1 and F2 are close, or in consonant portions. This can lead to jerky lip width movements. Future work will focus on improving formant extraction accuracy.
  • Consonant Lip Shape Estimation: Accurately estimating lip shape for consonants, especially bilabials like /p/, remains a challenge. The current method sometimes struggles to detect the necessary lip closure.
  • Lip Rounding on Robots: The inability of the current robot to perform lip rounding limits the full potential of the lip width control. Future work will likely involve developing robots with more advanced lip articulation capabilities.
  • Trade-off in Smoothing: Smoothing filters can prevent jerky motion but may also hinder the ability of the lips to fully close, as seen in bilabial consonants.
  • Subjective Naturalness Variation: While language independence was observed, the absolute naturalness judgment might vary among subjects with different origins, suggesting a need for further investigation into cultural perception of naturalness.

Summary updated

BibTeX
@inproceedings{ishi2012evaluation,
  author = {Carlos T. Ishi and Chaoran Liu and Hiroshi Ishiguro and Norihiro Hagita},
  title = {{Evaluation of a formant-based speech-driven lip motion generation}},
  booktitle = {Interspeech},
  year = {2012},
}
Figure