Adaptation techniques for speech synthesis in under-resourced languages

Gopala Krishna Anumanchipalli, Alan W. Black

TL;DR

This paper tackles the challenge of building high-quality speech synthesizers for languages with limited data. They propose and evaluate adaptation techniques within a statistical parametric speech synthesis framework, demonstrating that even with minimal target speaker or language data, acceptable synthesis quality can be achieved by leveraging existing resources and multilingual models.

What Problem Does This Paper Address?

Creating natural-sounding text-to-speech (TTS) systems typically requires extensive amounts of high-quality speech data and linguistic resources for each target language and speaker. This is a significant barrier for the vast majority of the world's languages, which are considered "under-resourced." The paper addresses two main scenarios: 1) building a voice for a new speaker when only a small amount of their speech is available, and 2) building a voice for a language with very little or no existing speech data or linguistic tools. The goal is to make TTS technology accessible for these under-resourced languages and speakers.

Methods

The authors utilize a Statistical Parametric Speech Synthesis (SPSS) framework called Clustergen. This approach models speech features (like spectral and pitch information) using clustered Gaussian distributions, which are then generated frame-by-frame. The key innovation lies in adaptation techniques applied to this framework:

  • Speaker Adaptation: When limited data from a target speaker is available, they adapt a pre-existing "source model" (built from a large, diverse dataset) to the target speaker's voice. They compare two adaptation methods:
  • MLLR (Maximum Likelihood Linear Regression): A model-based adaptation technique.
  • GMM-JDE (Joint Density GMM-based Estimation): A feature-based adaptation technique.
  • Multilingual Speech Synthesis: For languages with scarce data, they build a "multilingual voice" by training on speech data from multiple languages. This involves creating a global phoneme set and carefully mapping phonemes across languages. The multilingual voice then serves as a powerful "source model" that can be adapted to a specific target language with limited data.

Datasets

  • Arctic Database: Used for training the "average" American English voice and for evaluating speaker adaptation with limited target speaker data.
  • Wall Street Journal (WSJ) speech corpus: Provided data for training the initial "average" American English voice.
  • Globalphone database: A subset was selected to simulate limited new language data conditions, including speakers from Chinese (ZH), English (EN), German (DE), Japanese (JP), Croatian (KR), Portuguese (PT), Russian (RU), Spanish (ES), Swedish (SW), and Turkish (TR). German was used as a test target language.

Key Insights and Ideas

  • Leveraging "Average" Voices: An "average" voice built from many speakers is surprisingly effective as a source model for adapting to new speakers, as it captures general speech characteristics.
  • Multilingualism as a Resource Booster: Training a single model on data from multiple languages can significantly "boost" the phonetic coverage and quality for under-resourced languages, even if the target language itself has very little data.
  • Phonetic Similarity Matters: The choice of languages included in the multilingual training set is crucial. Removing languages that are phonetically similar to the target language can lead to worse adaptation results, suggesting that the models benefit from diverse phonetic information.
  • Data Quantity is Key: The quality of the synthesized speech, as measured by Mel-Cepstral Distortion (MCD), shows a strong, often linear, correlation with the amount of training data available, even in multilingual settings.

Key Results

  • Speaker Adaptation: With as few as 20 utterances from a target speaker, significant improvements in voice quality (reduction in MCD) were observed compared to the source model. MLLR adaptation showed slightly better performance with more data than GMM-JDE.
  • Multilingual Adaptation:
    • For German (DE), the synthesized voice quality was better than for Telugu (TE) even before adaptation, likely because German is phonetically closer to the European languages in the training set.
  • Adaptation with increasing amounts of target language data consistently improved voice quality for both German and Telugu, as indicated by decreasing MCD scores.
  • Experiments removing specific languages from the multilingual training set showed that removing English (phonetically similar to German) worsened German adaptation, while removing Chinese had less impact. For Telugu, removing English had little effect, but removing Chinese surprisingly improved results, suggesting complex interactions in phonetic representation.

Limitations and Future Work

  • Automatic Method Discovery: The authors acknowledge that they have "yet to discover an efficient automatic method to improve these existing techniques," implying that manual selection of languages and phonetic mappings is still required.
  • Phonetic Mapping Complexity: The process of creating a global phoneme set and mapping phonemes across languages requires careful development and can be non-trivial.
  • Future Directions: The paper suggests further research into more efficient automatic methods for language selection and phonetic mapping to further streamline the process of building voices for under-resourced languages. Exploring the impact of different acoustic phonetic feature weightings in phoneme similarity calculations could also yield improvements.

Summary updated

BibTeX
@inproceedings{anumanchipalli2010adaptation,
  author = {Gopala Krishna Anumanchipalli and Alan W. Black},
  title = {{Adaptation techniques for speech synthesis in under-resourced languages}},
  booktitle = {SLTU},
  year = {2010},
}
Figure