A German viseme-set for automatic transcription of input text used for audio-visual speech synthesis

Christian Weiss, Bianca AschenbernerInterspeech 2005

TL;DR

This paper introduces a novel German viseme set and an automatic transcription system for creating realistic audio-visual speech synthesis. By mapping phonemes to visual mouth shapes (visemes), the system enables more natural human-computer interaction and improves the intelligibility of synthesized speech.

What Problem Does This Paper Address?

Creating realistic audio-visual speech synthesis, where synthesized speech is accompanied by synchronized lip movements, is a challenging but important area for improving human-computer interfaces. While text-to-speech synthesis relies on phonetic transcriptions, visual synthesis requires a corresponding representation of mouth shapes, known as visemes. Existing research has focused on English, leaving a gap for German. This paper addresses the need for a comprehensive German viseme inventory and an automatic system to transcribe input text into these visemes, which is crucial for selecting appropriate visual speech segments for data-driven synthesis systems.

Methods

The authors developed a German viseme inventory by grouping phonemes based on their visual articulatory similarities. They then trained a Maximum Entropy model for automatic grapheme-to-viseme transcription. This approach leverages statistical methods to learn the mapping from written text to visual mouth shapes, aiming for high accuracy and efficiency. The viseme set was designed to be visually distinguishable by humans, even when phonemes might be acoustically similar.

Datasets

The paper utilizes the "BOSS II (DE)" phoneme inventory for German speech synthesis as a basis for their viseme mapping. While specific datasets for training the viseme transcription model are not explicitly detailed beyond the mention of "recorded speech corpora," the authors indicate that their viseme set is intended for use within their existing data-driven audio-visual synthesis system.

Key Insights and Ideas

  • Visemes are Crucial for Visual Realism: The paper highlights that visual cues, like lip movements, are as important as acoustic cues for speech perception and intelligibility, especially in human-machine interaction.
  • Phoneme-to-Viseme Mapping is Non-Trivial: Grouping phonemes into visemes requires careful consideration of visual distinguishability. For instance, while /p/ and /b/ are visually similar, the authors chose to separate /m/ due to its distinct lip closure in final positions. Similarly, acoustically distinct but visually similar sounds (like /t/, /d/, /k/, /g/) are grouped into a single viseme.
  • Maximum Entropy for Visemic Transcription: The use of Maximum Entropy models, a statistically motivated approach, proved effective for learning the complex mapping from text to visemes, achieving a respectable performance.
  • German-Specific Needs: The work directly addresses the lack of a standardized German viseme set, which is essential for developing German audio-visual speech synthesis systems.

Key Results

The paper reports achieving a 95.34% performance in automatic visemic transcription using their trained Maximum Entropy model. This indicates a high level of accuracy in converting input text into a sequence of visemes, which is a critical step for their audio-visual synthesis system. The authors state that this automatic transcription is already being used in their current system for selecting appropriate visual segments.

Limitations and Future Work

The authors acknowledge that their viseme inventory is based on a SAMPA-like labeling and that further modifications might be needed to address different needs, such as in audio-visual speech recognition. They also suggest that the developed system provides a good starting point for further research and refinement in the field of audio-visual speech synthesis. The paper implies that while the model performs well, continuous improvement and adaptation for specific applications are likely future directions.

Summary updated

BibTeX
@inproceedings{weiss2005german,
  author = {Christian Weiss and Bianca Aschenberner},
  title = {{A German viseme-set for automatic transcription of input text used for audio-visual speech synthesis}},
  booktitle = {Interspeech},
  year = {2005},
}
Figure