Adapt Mandarin TTS system to Chinese dialect TTS systems

Hai Ping Li, Wei ZhangInterspeech 2005

TL;DR

This paper presents a novel method for adapting existing Mandarin Text-to-Speech (TTS) systems to generate speech in various Chinese dialects. By leveraging phonological mappings and dialect-specific vocabularies, the authors significantly reduce the effort required to build dialect TTS systems, demonstrating success with Sichuan, Shaanxi, and Dongbei dialects.

What Problem Does This Paper Address?

Building high-quality Text-to-Speech (TTS) systems for individual Chinese dialects is a time-consuming and resource-intensive endeavor. Traditional methods require extensive data collection, phonological analysis, dictionary creation, and rule development for each dialect. This paper addresses the challenge of efficiently creating dialect TTS systems by proposing an adaptation approach that leverages the existing infrastructure of a well-developed Mandarin TTS system. The core problem is the high cost and effort associated with developing TTS for the diverse landscape of Chinese dialects, which have distinct phonologies and vocabularies despite a shared written language.

Methods

The proposed method adapts a Mandarin TTS system to dialect TTS systems through two primary mechanisms:

  1. Phonology Mapping: Dialect phonology is represented using Mandarin phonology. This involves creating mappings for dialect initials, finals, and tones to their Mandarin counterparts. This allows the core phonetic processing of the Mandarin TTS system to be reused.
  2. Dialect Vocabulary: A dialect-specific vocabulary is added to the Front End (FE) of the TTS system. This handles lexical differences, mapping dialect-specific words to their Mandarin equivalents to ensure correct pronunciation and understanding of dialectal phrases.

The authors utilize an IBM trainable TTS system as a base, which is a concatenative TTS system where units are syllables. The system construction is divided into off-line (unit set and prosody model generation) and run-time synthesis (FE and BE). The FE handles text analysis and symbolic prosody prediction, while the BE manages unit selection and modulation. The adaptation primarily modifies the FE by incorporating the dialect vocabulary and the phonology mappings. The symbolic prosody models are kept the same, assuming syntactic similarity between Mandarin and the target dialects.

Datasets

The paper describes the construction and evaluation of TTS systems for three specific Chinese dialects: Sichuan, Shaanxi, and Dongbei. These dialects are sub-dialects of the Beifang dialect.

  • Training/Construction: The paper mentions that dialect corpus recorded by professional native speakers was used to draw the phonology mappings. For evaluation, three scripts for the greeting domain were designed, each consisting of about 500 sentences selected from a larger corpus of over 20,000 greeting sentences. These scripts were used to create recordings by professional native dialect speakers.
  • Evaluation: The evaluation involved synthesized samples and recordings of the three dialects, as well as synthesized Mandarin samples. The corpus for the Mandarin TTS system used for comparison consisted of 5000 sentences from various domains.

Key Insights and Ideas

  • Leveraging Existing Infrastructure: The core idea is that by mapping dialect phonology and vocabulary to Mandarin, a significant portion of an existing Mandarin TTS system can be reused, drastically reducing development effort.
  • Dialect Suitability: Not all Chinese dialects are equally suitable for this adaptation method. The paper identifies that dialects with consistent phonology mappings and syntactic similarities to Mandarin are good candidates.
  • Importance of Phonological and Lexical Mapping: The success hinges on accurately mapping dialect sounds and words to their Mandarin equivalents. This allows the system to process dialectal input using familiar Mandarin phonetic rules and lexicons.
  • Domain-Specific Success: The method shows promise for specific domains like voice greetings, where dialectal variations are common and user interest is high.

Key Results

The paper reports on the evaluation of TTS systems for Dongbei, Shaanxi, and Sichuan dialects. The evaluation involved native dialect speakers and Mandarin speakers assessing the quality of synthesized dialect speech, synthesized Mandarin speech, and original dialect recordings.

  • Subjective Evaluation (MOS Scores):
    • Dialect MOS: The synthesized dialect systems achieved MOS scores of 2.70 (Dongbei), 3.54 (Shaanxi), and 3.11 (Sichuan).
  • Mandarin MOS: The synthesized Mandarin system achieved MOS scores of 3.57 (Dongbei evaluators), 3.78 (Shaanxi evaluators), and 3.67 (Sichuan evaluators).
  • Comparison: The scores for dialect recordings were generally higher (4.21, 4.43, 3.94) than synthesized dialect samples, indicating room for improvement in synthesis quality. However, the synthesized dialect samples were rated acceptably (2.27, 3.14, 2.46). The synthesized Mandarin samples were rated well (3.54, 3.78, 3.64).
  • Efficiency: The method significantly saves effort compared to traditional methods of building dialect TTS systems from scratch.

Limitations and Future Work

  • Dialect Suitability: The paper explicitly states that not all Chinese dialects are suitable for this adaptation method. Cantonese (Yue dialect), Shanghaiese (Wu dialect), Kejia, and Min dialects are mentioned as unsuitable because it is difficult or impossible to generate consistent phonology mappings.
  • Syntactic Differences: The current method does not account for syntactic differences between Mandarin and dialects, assuming they are not significant, especially for written language.
  • Future Work: The authors suggest that more dialects are under investigation to apply this method. Further research could explore how to handle more significant syntactic variations and improve the naturalness of synthesized dialect speech.

Summary updated

BibTeX
@inproceedings{li2005adapt,
  author = {Hai Ping Li and Wei Zhang},
  title = {{Adapt Mandarin TTS system to Chinese dialect TTS systems}},
  booktitle = {Interspeech},
  year = {2005},
}
Figure