Abstract
In Mandarin text-to-speech (TTS) systems, the front-end text processing module significantly influences the intelligibility and naturalness of synthesized speech. Building a typical pipeline-based front-end, which consists of multiple individual components, requires extensive effort. In this paper, we propose a unified sequence-to-sequence front-end model for Mandarin TTS that converts raw texts to linguistic features directly. Compared to the pipeline-based front-end, our unified front-end achieves comparable performance in polyphone disambiguation and prosody word prediction, and improves intonation phrase prediction by 0.0738 in F1 score. We also implemented the unified front-end with Tacotron and WaveRNN to build a Mandarin TTS system. The synthesized speech achieved a Mean Opinion Score (MOS) of 4.38, comparable with the pipeline-based front-end (4.37) and close to human recordings (4.49).
Introduction
The front-end of a Text-to-Speech (TTS) system is responsible for converting raw text into linguistic features, including phonemes, tones, and prosodic markers. For Mandarin, this process is particularly challenging due to the lack of explicit word boundaries (requiring Chinese Word Segmentation or CWS), the prevalence of polyphones (characters with multiple pronunciations), and the complexity of prosody prediction.
Traditional Mandarin TTS front-ends rely on a pipeline of independent models. However, this approach suffers from error propagation, where a mistake in an early stage (like CWS) negatively impacts downstream tasks like part-of-speech (POS) tagging and pronunciation. Furthermore, maintaining multiple separate models increases engineering complexity.
Proposed Method
We propose a unified sequence-to-sequence (seq2seq) front-end model that predicts phonemes, tones, and prosody levels simultaneously from raw text.
Architecture
The core architecture is an encoder-decoder model utilizing Gaussian Mixture Model (GMM) attention. GMM attention is selected for its superior ability to maintain monotonicity, ensuring the model processes text in a linear sequence without skipping or repeating characters.
Auxiliary Module and Joint Modeling
To leverage the benefits of word segmentation and POS information without the fragility of a hard pipeline, we incorporate an auxiliary module. This module extracts latent representations of CWS and POS to guide the main model. The system is trained to jointly predict phonemes, tones, and three levels of prosody: Prosody Word (PW), Prosody Phrase (PP), and Intonation Phrase (IP).
Semi-Auto Regressive (SAR) Inference
To improve inference stability, we employ a Semi-Auto Regressive (SAR) strategy. Since most Mandarin characters have a single pronunciation, the model uses auto-regressive logic primarily for polyphones, while using known correct phonemes for standard characters to prevent error accumulation.
Experimental Results
The model was trained using the People’s Daily Corpus for the auxiliary module and an internal corpus of 118,000 utterances for the unified model. Character embeddings were pre-trained on Wikipedia using Word2Vec.
Performance Metrics
The unified model achieved a G2P accuracy of 96.56% in polyphone disambiguation. Notably, it significantly outperformed the baseline in Intonation Phrase (IP) prediction, with an F1 score improvement of 0.0738, suggesting that the seq2seq structure effectively captures long-range sentence context.
Speech Quality
To evaluate the final speech quality, the front-end was integrated with Tacotron and WaveRNN. Mean Opinion Score (MOS) tests were conducted with human listeners to compare the unified system (U-system) against a traditional pipeline (P-system) and ground truth recordings.
As shown in Figure 5, the U-system achieved a MOS of 4.38, which is virtually identical to the P-system (4.37) and very close to actual human recordings (4.49).
Conclusion
This work demonstrates that a unified seq2seq model can replace complex pipeline-based front-ends for Mandarin TTS without sacrificing performance. By jointly modeling linguistic features and utilizing latent auxiliary information, the system simplifies the TTS workflow while maintaining high speech naturalness. While the SAR inference method can struggle with consecutive polyphones, the overall performance is robust. Future work will explore the use of BERT embeddings for deeper semantic context and extending the architecture to other languages.
Summary updated