Setup for acoustic-visual speech synthesis by concatenating bimodal units

Asterios Toutios, Utpala Musti, Slim Ouni, Vincent Colotte, Brigitte Wrobel-Dautcourt, Marie-Odile BergerInterspeech 2010

TL;DR

This paper introduces a novel approach to audiovisual speech synthesis by treating acoustic and visual speech data as a single "bimodal" signal. By concatenating bimodal units (diphones that include both sound and 3D facial motion data), the system aims to overcome synchronization and coherence issues inherent in traditional methods, leading to more natural and realistic talking head animations.

What Problem Does This Paper Address?

Traditional audiovisual speech synthesis systems often generate speech acoustics and facial animations separately and then attempt to synchronize them. This approach faces significant challenges: achieving perfect synchronization is difficult, and even when synchronized, the auditory and visual streams might originate from different recordings, leading to perceptual incoherence. The authors argue that speech is inherently bimodal, with strong correlations between vocal tract movements and the resulting sound. They propose to treat these two modalities as a unified signal from the outset to ensure intrinsic consistency and improve the overall quality of synthesized speech and facial animation.

Methods

The core of the proposed method is the concatenation of "bimodal diphones." These units are not just acoustic speech segments but also contain corresponding 3D facial motion data.

  • Data Acquisition: A low-cost stereovision system was used to capture 3D facial data (positions of 252 markers) simultaneously with audio. This data was acquired at a high sampling rate (188.27 Hz) for 319 French sentences, totaling about 25 minutes of speech.
  • Visual Data Modeling: Principal Component Analysis (PCA) was applied to the facial marker data, focusing on the lower part of the face (jaw, lips, cheeks) which is most relevant to speech articulation. The first 12 principal components were found to explain about 94% of the variance, and these reduced representations are used as the visual component of the bimodal units.
  • Bimodal Unit Selection: Similar to standard concatenative speech synthesis, the system phonetizes the input text and partitions it into diphones. A database of bimodal diphones is constructed. Unit selection is performed using a Viterbi algorithm that minimizes a weighted combination of three costs:
  • Target Cost (TC): Based on linguistic analysis of the target utterance.
  • Acoustic Join Cost (JC): Measures acoustic similarity at the boundaries of concatenated units.
  • Visual Join Cost (VC): Measures the visual similarity (distance between PCA-reduced facial trajectories) at the boundaries of concatenated units.
  • Concatenation: Acoustic concatenation uses traditional techniques with pitch adjustments. Visual trajectories are currently concatenated directly without explicit processing at boundaries, relying on the inherent continuity of the PCA-reduced data and the visual join cost.

Datasets

The primary dataset used for this work consists of approximately 25 minutes of speech from 319 medium-sized French sentences, uttered by a single male speaker. This dataset includes synchronized audio and 3D facial marker data. The authors mention plans to expand this to a corpus of about 2000 sentences (over 2 hours of speech).

Key Insights and Ideas

  • Unified Bimodal Signal: The fundamental idea of treating acoustic and visual speech as a single, inseparable signal is key. This contrasts with typical approaches that handle them as independent streams.
  • Visual Join Cost: Introducing a specific cost function for visual continuity at diphone boundaries is crucial for ensuring smooth facial animations.
  • PCA for Visual Representation: Using PCA to reduce the dimensionality of facial motion data makes it more manageable and allows for efficient integration into the cost function for unit selection.
  • Benefits of Bimodality: Preliminary results suggest that incorporating visual constraints significantly improves the continuity of facial animations, reducing "jerks" and visual discontinuities that plague acoustic-only synthesis.

Key Results

  • Improved Visual Continuity: When the visual join cost was minimized (either alone or in combination with acoustic costs), the synthesized visual trajectories showed significantly greater continuity compared to an acoustic-only synthesis approach. This directly translates to smoother facial animations.
  • Acoustic Quality: The bimodal synthesis resulted in an acoustic waveform that was much closer to the acoustic-only synthesis output than to the visual-only synthesis output. This indicates that the bimodal approach preserves good acoustic quality while enhancing visual realism.
  • Potential for Coarticulation: The system showed a tendency to select more consecutive diphones from the database when visual constraints were included, suggesting that the visual information might help resolve ambiguities or improve the selection of units that exhibit better coarticulatory transitions.

Limitations and Future Work

  • Corpus Size: The current corpus of 25 minutes is considered small, and the authors believe a larger corpus (aiming for over 2 hours) will significantly improve results, especially for capturing visual variability.
  • Weight Optimization: The weights for the target, acoustic join, and visual join costs (wtc, wjc, wvc) were set using initial heuristics. Further exploration and refinement of these weights are planned.
  • Visual Trajectory Processing: While the current system concatenates visual trajectories directly, the authors acknowledge that explicit processing might be needed to impose continuity, especially for less important principal components.
  • Higher Resolution Meshes: The current visual output uses sparse meshes, which will be mapped to higher-resolution meshes in future work to enhance realism.
  • Generalization: The current work is based on a single speaker. Extending to multiple speakers and diverse speaking styles would be a significant next step.

Summary updated

BibTeX
@inproceedings{toutios2010setup,
  author = {Asterios Toutios and Utpala Musti and Slim Ouni and Vincent Colotte and Brigitte Wrobel-Dautcourt and Marie-Odile Berger},
  title = {{Setup for acoustic-visual speech synthesis by concatenating bimodal units}},
  booktitle = {Interspeech},
  year = {2010},
}
Figure