Speaker adaptation using a parallel phone set pronunciation dictionary for Thai-English bilingual TTS

Anocha Rugchatjaroen, Nattanun Thatphithakkul, Ananlada Chotimongkol, Ausdang Thangthai, Chai WutiwiwatchaiInterspeech 2009

TL;DR

This paper tackles the challenge of creating a single Text-to-Speech (TTS) system that can fluidly switch between Thai and English. They achieve this by adapting an English TTS voice to sound more like a Thai TTS voice, using a novel parallel phone set pronunciation dictionary to bridge the gap between the two languages' phonetic systems. This approach significantly improves the intelligibility of synthesized English words within a Thai context.

What Problem Does This Paper Address?

In our increasingly globalized world, digital content often mixes languages. This presents a challenge for Text-to-Speech (TTS) systems, which are typically designed for a single language. Imagine reading a Thai news article that suddenly includes an English product name or a foreign phrase – a standard Thai TTS system would struggle to pronounce the English part naturally, and vice-versa. Building a high-quality bilingual TTS system is difficult because it requires a large corpus of speech from a bilingual speaker, which is rare and expensive to create. Existing methods for handling mixed-language speech often rely on converting English words into Thai phonemes, but this leads to inaccurate pronunciations and reduced intelligibility. This paper aims to solve this by developing a Thai-English bilingual TTS system that can synthesize English words with high intelligibility and a more natural voice transition.

Methods

The core of this work lies in two main techniques:

  • Speaker Adaptation: The researchers start with two separate TTS systems: one for Thai and one for English, both built using the HMM-based TTS (HTS) framework. The problem is that the voices from these two systems sound distinct. To address this, they employ a speaker adaptation technique called CSMAPLR (Constrained Structural Maximum A Posteriori Linear Regression). This method adjusts the parameters of the English TTS system to make its synthesized voice sound more like the Thai TTS voice.
  • Cross-Language Pronunciation Mapping via Parallel Phone Set Dictionary: A major hurdle in adapting between languages is their differing phonetic inventories. The paper introduces a novel approach using a "parallel phone set pronunciation dictionary." This dictionary contains English words, each with two phonetic transcriptions: one using the Thai phone set and another using the English phone set. This allows them to generate adaptation data by having the Thai TTS system pronounce English words using their Thai phonetic transcriptions. These synthesized utterances are then labeled with the corresponding English phonetic transcriptions. This "cross-language pronunciation mapping" enables adaptation without needing explicit, manual mapping between Thai and English phones or a bilingual corpus.

Datasets

  • Thai Speech Synthesis Corpus 1 (TSynC-1): Used to train the Thai HMM-based TTS system. It's a single female speaker corpus with 13 hours of read speech.
  • CMU ARCTIC Databases: Used to train the initial English HMM-based TTS system. This involved training on data from 4 speakers using speaker-independent and speaker-adaptive methods.
  • Parallel Phone Set Pronunciation Dictionary: This is a newly introduced benchmark in the paper. It contains English words with both Thai and English phonetic transcriptions. This dictionary is crucial for generating the adaptation data.

Key Insights and Ideas

  • Bridging Phonetic Gaps: The parallel phone set dictionary is a clever way to overcome the fundamental differences between Thai and English phoneme sets, enabling adaptation without explicit phone mapping.
  • Adaptation Data Generation: Synthesizing English words with Thai pronunciations using the Thai TTS and then relabeling them with English pronunciations is an effective strategy for creating adaptation data.
  • Speaker Adaptation's Impact: While speaker adaptation improves voice similarity, it can subtly affect intelligibility, suggesting a trade-off that needs careful management.
  • Intelligibility vs. Pronunciation Accuracy: Even with correct Thai pronunciations of English words, intelligibility can be lower than using an adapted English TTS. This highlights the importance of using an appropriate phonetic representation for the target language.

Key Results

  • Improved Intelligibility: The proposed bilingual TTS system (BML_A) achieved a higher Mean Opinion Score (MOS) for intelligibility (0.61 higher) compared to the existing system that relies solely on Thai pronunciations of English words (ML).
  • Similar Intelligibility to Oracle: When English words were perfectly converted to Thai phonemes (ML_D), the intelligibility was comparable to the proposed system. However, achieving such perfect conversion is difficult in practice (around 62% accuracy).
  • Enhanced Similarity: The speaker adaptation method made the synthesized English words sound more similar to the synthesized Thai voice.
  • Trade-off Observed: The adapted English HTS (BML_A) showed slightly lower intelligibility than the non-adapted English HTS (BML) when the English G2P was perfect (ML_D). This is attributed to the quality of the adaptation data.

Limitations and Future Work

  • Adaptation Data Quality: The paper acknowledges that the quality of the adaptation data (synthesized English words with Thai pronunciations) can impact the intelligibility of the adapted English voice.
  • Similarity Improvement: While similarity was improved, the authors note there is still room for further enhancement.
  • Future Work: The authors plan to create a speech corpus that is parallel to the CMU ARCTIC but with a Thai accent. This would likely improve the adaptation performance by providing more natural Thai-accented English speech data. They also aim to improve the quality of the adaptation data itself.

Summary updated

BibTeX
@inproceedings{rugchatjaroen2009speaker,
  author = {Anocha Rugchatjaroen and Nattanun Thatphithakkul and Ananlada Chotimongkol and Ausdang Thangthai and Chai Wutiwiwatchai},
  title = {{Speaker adaptation using a parallel phone set pronunciation dictionary for Thai-English bilingual TTS}},
  booktitle = {Interspeech},
  year = {2009},
}
Figure