Pre-Trained Text Representations for Improving Front-End Text Processing in Mandarin Text-to-Speech Synthesis

Bing Yang, Jiaqi Zhong, Shan LiuInterspeech 2019 · DOI

Researchers at Tencent’s Cloud and Smart Industries Group have developed a modernized approach to the "front-end" of Mandarin Text-to-Speech (TTS) systems. By utilizing powerful pre-trained text representations like BERT and Neural Machine Translation (NMT) encoders, the team has demonstrated significant improvements in handling the intricate nuances of Chinese pronunciation and rhythm.

The Mandarin TTS Front-End Challenge

In a TTS system, the front-end acts as a translator, converting raw text into a detailed linguistic blueprint—including phonemes, pauses, and stress—that the acoustic model then synthesizes into audio. Mandarin presents two primary challenges for this stage:

  1. Polyphone Disambiguation: Many Chinese characters have multiple pronunciations (polyphones) depending on their context. Incorrect selection leads to unintelligible speech.
  2. Prosodic Structure Prediction: Natural-sounding Mandarin requires a precise hierarchy of pauses, moving from Prosodic Words (PW) to Prosodic Phrases (PP) and Intonation Phrases (IP).
The traditional Mandarin TTS front-end pipeline, highlighting the multi-stage process from raw text to prosodic structure and phoneme conversion.

Historically, these tasks relied on manual feature engineering (such as Part-of-Speech tags) or basic Recurrent Neural Networks (RNNs). These traditional methods often struggle with long-term context and require vast amounts of expensive, task-specific labeled data.

Methodology: Leveraging Pre-trained Representations

The core innovation of this research is the application of transfer learning. Instead of training models from scratch on limited TTS datasets, the authors employ "feature extractors" pre-trained on massive general-text corpora.

The proposed architecture integrating BERT and NMT-based pre-trained representations into the TTS front-end workflow.

The researchers evaluated two Transformer-based architectures:

  • BERT: An unsupervised model trained on the Chinese Wikipedia Corpus to capture deep bidirectional context.
  • NMT Encoder: The encoder portion of a supervised Neural Machine Translation model, trained on over 100 million Chinese-English sentence pairs to represent semantic meaning.

For the downstream tasks, a compact Multi-Layer Perceptron (MLP) was used for polyphone disambiguation, while a Bidirectional LSTM with Conditional Random Fields (BLSTM-CRF) handled prosodic prediction. The team also introduced BERT-MT, a multi-task model capable of predicting multiple prosodic levels simultaneously, increasing computational efficiency.

Key Insights and Ideas

  • The Power of Context: Unlike traditional RNNs, the multi-head self-attention mechanism in Transformers allows the model to process an entire sentence simultaneously. This global view is crucial for capturing the subtle semantic cues necessary to distinguish between different pronunciations of the same character.
  • Semantic Clustering: t-SNE visualizations revealed that BERT naturally clusters different pronunciations of polyphonic characters into distinct groups within its latent space. This inherent organization significantly simplifies the final classification task for the downstream MLP.
  • Supervised vs. Unsupervised Effectiveness: Both the unsupervised BERT and supervised NMT features performed exceptionally well. While BERT's deeper architecture gave it a slight edge, the NMT encoder demonstrated that translation-based semantic features are highly effective for TTS tasks.

Key Results

The proposed methods yielded substantial gains over traditional Maximum Entropy and standard Bi-LSTM baselines:

  • Polyphone Disambiguation: Accuracy reached 96.80% with BERT and 96.94% using a feature ensemble, a marked improvement over the 91.34% achieved by traditional statistical methods.
  • Prosodic Prediction: The models achieved F1 scores of 0.974 for Prosodic Word (PW) prediction and 0.851 for Prosodic Phrase (PP) prediction, representing absolute improvements of 0.013 and 0.027, respectively.

Conclusion and Future Directions

This research demonstrates that pre-trained representations can significantly enhance the naturalness and accuracy of Mandarin TTS. Future work will explore applying these representations to other front-end modules, such as text normalization, and investigating speaking style prediction to create more expressive and emotional synthesized voices. While the computational cost of large Transformer models remains a challenge for real-time production, these results provide a clear path forward for high-quality speech synthesis.

Summary updated

BibTeX
@inproceedings{yang2019pre,
  author = {Bing Yang and Jiaqi Zhong and Shan Liu},
  title = {{Pre-Trained Text Representations for Improving Front-End Text Processing in Mandarin Text-to-Speech Synthesis}},
  booktitle = {Interspeech},
  year = {2019},
  doi = {10.21437/interspeech.2019-1418},
}
Figure