Empowering Global Voices: A Data-Efficient, Phoneme-Tone Adaptive Approach to High-Fidelity Speech Synthesis

Yizhong Geng, Jizhuo Xu, Zeyu Liang, Jinghan Yang, Xiaoyi Shi, Xiaoyu ShenarXiv 2025 · DOI · arXiv

TL;DR

This paper introduces a novel, data-efficient approach to Text-to-Speech (TTS) synthesis for low-resource languages, specifically demonstrating its effectiveness with Thai. By combining a sophisticated preprocessing pipeline with a phoneme-tone adaptive acoustic model, they achieve state-of-the-art performance, enabling zero-shot voice cloning and industrial-grade applications even with limited data.

What Problem Does This Paper Address?

While TTS technology has made remarkable strides for widely spoken languages, a significant digital divide persists for the thousands of languages with limited available data. This is particularly challenging for linguistically complex languages like Thai, which features intricate tonal distinctions and ambiguous word boundaries. Existing advanced TTS models, especially those leveraging large language models (LLMs), are often too data-intensive to be practical for these under-resourced scenarios. This paper aims to bridge this gap by developing a high-fidelity TTS system that is both data-efficient and capable of handling the unique linguistic complexities of languages like Thai, thereby democratizing access to speech synthesis technology.

Methods

The proposed approach consists of two main components: a Data-Optimized Preprocessing Pipeline and an Advanced Acoustic Model.

The Preprocessing Pipeline tackles Thai's linguistic challenges head-on:

  • LLM for Pause Prediction: A pretrained LLM is fine-tuned to predict prosodic pauses in unpunctuated Thai text, crucial for natural speech flow.
  • Tokenizer: An enhanced tokenizer, augmented with a comprehensive Thai word corpus, segments unspaced Thai orthography into words, improving accuracy.
  • Grapheme-to-Phoneme (G2P) Converter: This module maps Thai characters to IPA phonemes, crucially incorporating tone markers. It uses a hybrid approach combining rule-based alignment and a transformer model for ambiguous cases, leveraging a dedicated Phoneme-Tone Annotation Dataset.

The Advanced Acoustic Model then synthesizes speech:

  • Pretrained Feature Extractors: Models pretrained on multilingual datasets extract duration, pitch, and energy from audio. These are fine-tuned on Thai data to adapt to the low-resource setting.
  • Phoneme-Tone BERT: This module encodes the phoneme-tone sequence, providing contextual prosody information.
  • GAN-based Decoder: This decoder generates waveforms directly from phoneme sequences, duration, pitch, energy, and style vectors, optimizing for reconstruction loss in time, frequency, and perceptual domains.
  • Style Encoder: This module extracts a style vector from the input waveform, enabling zero-shot voice cloning.

The prediction module (duration, pitch, energy) is initially trained independently and then co-trained with the decoder for improved synthesis quality.

Datasets

The paper introduces a comprehensive, multi-dimensional Thai TTS dataset constructed to support industrial-scale synthesis under low-resource conditions. It comprises:

  • Speech Dataset:
    • Multi-domain Corpus: 500 hours of speech from diverse sources (News Broadcasts, Audiobooks, Social Media Short Videos, Daily Conversation Podcasts).
  • Vertical Domain Dataset: 40 hours of speech covering specialized fields (Finance, Healthcare, Education, Law).
  • Thai Text Dataset:
    • Sentence Corpus: 1,000,000 sentences for training the Phoneme-Tone Bert module.
  • Word Corpus: 100,000 words to train the tokenizer.
  • Annotation Dataset:
    • Pause Annotation: 15,000 sentences manually annotated for prosodic boundaries.
  • Phoneme-Tone Annotation: 40,000 words with detailed IPA phoneme and tone markings for G2P conversion and tone modeling.

Key Insights and Ideas

  • Data Efficiency is Key: The core insight is that sophisticated preprocessing and adaptive modeling can compensate for limited data, making high-quality TTS accessible for under-resourced languages.
  • Phoneme-Tone Integration is Crucial for Thai: Explicitly modeling tones and their interaction with phonemes is vital for accurate and natural Thai speech synthesis, as even minor tone shifts alter meaning.
  • Zero-Shot Voice Cloning is Achievable: The proposed style encoder allows for voice cloning without speaker-specific fine-tuning, significantly reducing deployment time and effort for new voices.
  • Domain-Specific Adaptation Matters: The inclusion of a vertical domain dataset demonstrates the system's ability to handle specialized vocabulary and jargon accurately, crucial for industrial applications.
  • Ablation Studies Highlight Module Importance: The paper effectively shows that each component of the preprocessing pipeline (pause prediction, tokenization, G2P) contributes significantly to overall performance, with G2P having the most substantial impact.

Key Results

  • State-of-the-Art Performance: The proposed system achieves the highest overall accuracy and speech quality among open-source systems on a general-domain test set, with a Word Error Rate (WER) of 6.3%.
  • Domain-Specific Robustness: The system demonstrates strong performance in specialized domains (finance, healthcare, education, law), outperforming proprietary solutions in these contexts.
  • Superior Zero-Shot TTS: The system achieves a Similarity (SIM) score of 0.91 and a Subjective Mean Opinion Score (SMOS) of 4.5, surpassing OpenVoice (0.85 SIM, 4.0 SMOS) in zero-shot voice cloning.
  • Significant Improvement from Preprocessing: Ablation studies show that removing pause optimization, tokenization, or G2P significantly degrades performance, with G2P optimization yielding the largest gains.

Limitations and Future Work

  • Limitations: While the paper focuses on Thai, the authors acknowledge that the system's performance might vary for languages with different linguistic structures. The reliance on specific pretrained models might also introduce dependencies.
  • Future Work: The authors propose extending this data-efficient framework to other under-resourced languages with similar constraints. Further research could explore more advanced LLM integration for even richer contextual understanding and prosody modeling.

Summary updated

BibTeX
@misc{geng2025empowering,
  author = {Yizhong Geng and Jizhuo Xu and Zeyu Liang and Jinghan Yang and Xiaoyi Shi and Xiaoyu Shen},
  title = {{Empowering Global Voices: A Data-Efficient, Phoneme-Tone Adaptive Approach to High-Fidelity Speech Synthesis}},
  year = {2025},
  eprint = {2504.07858},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2504.07858},
}
Figure