A Unified Front-End Framework for English Text-to-Speech Synthesis

Zelin Ying, Chen Li, Yu Dong, Qiuqiang Kong, Qiao Tian, Yuanyuan Huo, Yuxuan Wang

TL;DR

This paper introduces a novel unified framework for the English text-to-speech (TTS) front-end, integrating text normalization (TN), prosody prediction (PWPP), and grapheme-to-phoneme (G2P) conversion into a single, multi-task learning system. This approach significantly improves performance across all modules by capturing their interdependencies, achieving state-of-the-art results.

What Problem Does This Paper Address?

The front-end of a Text-to-Speech (TTS) system is crucial for converting raw text into linguistic features that a speech synthesis model can use to generate natural-sounding speech. Traditionally, the English TTS front-end is broken down into three main modules: Text Normalization (TN) to convert numbers, abbreviations, etc., into spoken words; Prosody Word Prosody Phrase (PWPP) to predict pauses and intonation; and Grapheme-to-Phoneme (G2P) to convert text into phonetic representations.

While individual modules have seen significant research, current approaches often treat them in isolation. This separation neglects the inherent dependencies between these modules, leading to suboptimal performance for each. For instance, accurate phoneme prediction (G2P) can be hindered by incorrect text normalization, and prosody prediction (PWPP) is influenced by both normalized text and phonetic information. This paper addresses this fragmentation by proposing a unified framework that leverages multi-task learning to jointly optimize these critical front-end components.

Methods

The core of this paper is a unified multi-task learning framework that integrates TN, PWPP, and G2P modules. Instead of training each module independently, they are trained together using a shared BERT model as the language encoder. This shared model is fine-tuned across five tasks: TN, PWPP, G2POOV (G2P for Out-Of-Vocabulary words), POS (Part-of-Speech tagging for homograph disambiguation), and Polyphone (handling homographs with the same POS but different pronunciations).

The framework employs a hybrid approach for TN, combining rule-based methods with a neural model to handle complex cases and "hot words" effectively. For PWPP, a hierarchical sequence tagging structure is used, where each prosody level is predicted independently but with consideration for constraints from higher levels, leading to better overall prosody prediction. The G2P module is enhanced by incorporating POS and Polyphone tasks to specifically address homograph disambiguation, a common challenge in English. The G2POOV task handles words not found in the lexicon.

Datasets

  • TN Module: Google's open-source English text normalization dataset (755,441 sentences).
  • PWPP Module: An internal dataset (100,026 sentences).
  • G2P Module:
    • POS task: 102,164 sentences.
  • Polyphone task: 28,097 sentences.
  • G2POOV task: Trained using a dictionary.
  • Real-world test set: 2560 sentences for evaluating the complete G2P module.

The paper does not introduce new datasets but utilizes existing ones for evaluation.

Key Insights and Ideas

  • Interdependence is Key: The most significant insight is that treating TN, PWPP, and G2P as separate entities leads to sub-optimal performance. A unified approach that captures their interdependencies yields substantial improvements.
  • Multi-Task Learning Synergy: Jointly training these tasks allows the model to learn shared representations and leverage information across modules, leading to better generalization and accuracy.
  • Hybrid TN for Robustness: Combining rule-based systems with neural models for text normalization provides a robust solution that can handle both general cases and specific exceptions effectively.
  • Hierarchical PWPP for Nuance: The hierarchical approach to prosody prediction better captures the complex relationships between different levels of prosodic structure, leading to more accurate pause and intonation predictions.
  • Targeted Homograph Resolution: Explicitly addressing homographs through POS and Polyphone tasks within the G2P module is crucial for accurate phonetic conversion in English.

Key Results

The proposed unified framework achieved state-of-the-art (SOTA) performance across all evaluated modules:

  • TN Module: Achieved a Sentence Error Rate (SER) of 1.19%, outperforming previous SOTA methods like Seq2Edits (1.36%).
  • PWPP Module: Demonstrated significant improvements with hierarchical sequence tagging, achieving F1-scores of 90.83% (#1), 57.65% (#2), and 83.36% (#3) for the different prosody levels, notably outperforming traditional methods.
  • G2POOV Task: Achieved a Word Error Rate (WER) of 19.42%, surpassing previous SOTA methods like r-G2P (19.85%).
  • Complete G2P Module: Achieved a WER of 3.09% on real-world sentences, showcasing the effectiveness of the integrated G2POOV, POS, and Polyphone tasks.

Limitations and Future Work

  • Language Specificity: While the paper focuses on English, the authors note that unified front-end frameworks for other languages cannot be directly compared due to language-specific differences. Future work could explore extending this unified approach to other languages.
  • Data Availability: The PWPP module evaluation used an internal dataset. While the results are promising, the availability and characteristics of this dataset are not fully detailed.
  • Further Integration: The paper successfully unifies the core front-end tasks. Future research could explore integrating these front-end modules even more deeply with the acoustic model for end-to-end TTS systems.
  • Error Analysis: While quantitative results are strong, a deeper qualitative analysis of the types of errors still made by the unified framework, especially in challenging cases, could inform future improvements.

Summary updated

Abstract

The front-end is a critical component of English text-to-speech (TTS) systems, responsible for extracting linguistic features that are essential for a text-to-speech model to synthesize speech, such as prosodies and phonemes. The English TTS front-end typically consists of a text normalization (TN) module, a prosody word prosody phrase (PWPP) module, and a grapheme-to-phoneme (G2P) module. However, current research on the English TTS front-end focuses solely on individual modules, neglecting the interdependence between them and resulting in sub-optimal performance for each module. Therefore, this paper proposes a unified front-end framework that captures the dependencies among the English TTS front-end modules. Extensive experiments have demonstrated that the proposed method achieves state-of-the-art (SOTA) performance in all modules.
BibTeX
@inproceedings{ying2024unified,
  author = {Zelin Ying and Chen Li and Yu Dong and Qiuqiang Kong and Qiao Tian and Yuanyuan Huo and Yuxuan Wang},
  title = {{A Unified Front-End Framework for English Text-to-Speech Synthesis}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2024},
  doi = {10.1109/icassp48485.2024.10447144},
}
Figure