This paper, a collaboration between Bytedance AI-Lab and Shanghai Jiaotong University, presents a streamlined approach to one of the most complex parts of speech synthesis: the Mandarin text-to-speech (TTS) front-end.
TL;DR
Researchers have developed a unified sequence-to-sequence model that replaces the traditional, error-prone "pipeline" of Mandarin text processing. By predicting phonemes, tones, and prosody simultaneously, the system simplifies the TTS workflow while achieving speech naturalness nearly indistinguishable from human recordings.
What Problem Does This Paper Address?
In TTS, the "front-end" is responsible for turning raw text into linguistic features (how words are pronounced and where the pauses go). For Mandarin, this is notoriously difficult. Unlike English, Mandarin requires:
- Chinese Word Segmentation (CWS): Identifying word boundaries in a continuous string of characters.
- Polyphone Disambiguation: Choosing the right pronunciation for characters with multiple sounds (e.g., "行" can be xíng or háng).
- Prosody Prediction: Determining the rhythm and intonation (pauses at word, phrase, and sentence levels).
Traditionally, this is done via a pipeline: a series of independent models where the output of one feeds the next. The problem? Error propagation. If the word segmenter makes a tiny mistake, it ruins the part-of-speech tagger, which ruins the pronunciation, leading to "robotic" or incorrect speech. Furthermore, maintaining five or six separate models is an engineering nightmare.
Methods
The authors propose a Unified Sequence-to-Sequence Front-End. Instead of a chain of models, they use a single architecture to handle everything.
- The Architecture: An encoder-decoder model using GMM (Gaussian Mixture Model) Attention. This type of attention is specifically chosen because it excels at "monotonicity"—ensuring the model processes the text in the correct linear order without skipping or repeating words.
- Auxiliary Module: To keep the benefits of word segmentation and part-of-speech (POS) data without the pipeline's fragility, they include an auxiliary module (testing both Dilated CNNs and Transformers). This module extracts "latent representations" of CWS and POS to guide the main model.
- SAR (Semi-Auto Regressive) Inference: This is a clever inference trick. Since most Mandarin characters have only one pronunciation, the model only uses its "auto-regressive" (step-by-step) logic for polyphones. For standard characters, it uses the known correct phoneme to prevent small errors from snowballing.
- Joint Modeling: The model is trained to predict phonemes, tones, and three levels of prosody (Prosody Word, Prosody Phrase, and Intonation Phrase) all at once.
Datasets
- People’s Daily Corpus: 940,000 Mandarin utterances used to train the auxiliary module for word segmentation and POS tagging.
- Internal Corpus: Approximately 118,000 utterances with high-quality prosody and Pinyin labels for training the unified model.
- Wikipedia (1Gb): Used to pre-train 300-dimensional character embeddings via Word2Vec.
Key Insights and Ideas
- Simplification without Sacrifice: The biggest "aha" moment is that a single joint model can perform as well as a complex pipeline. By predicting everything together, the model learns the inherent relationships between word boundaries and intonation.
- The Power of Latent Features: By using CWS and POS as auxiliary inputs rather than hard constraints, the model becomes more robust. It "looks" at the word boundaries but isn't strictly forced to follow a potentially buggy segmentation.
- Intonation Breakthrough: The unified model significantly outperformed the baseline in Intonation Phrase (IP) prediction (an F1 score boost of 0.0738). This suggests that the seq2seq structure captures long-range sentence context better than a serial pipeline.
Key Results
- Speech Quality: In Mean Opinion Score (MOS) tests—where humans rate speech naturalness—the unified system (U-system) scored 4.38. This is virtually identical to the complex pipeline (4.37) and very close to actual human recordings (4.49).
- G2P Accuracy: The model achieved 96.56% accuracy in polyphone disambiguation, outperforming several standalone baseline models.
- Efficiency: The researchers successfully integrated this front-end with Tacotron and WaveRNN, proving it works in a full, end-to-end production-style environment.
Limitations and Future Work
- Continuous Polyphones: The SAR inference method can struggle when a sentence contains many polyphones in a row; an error on the first one can still lead to errors on the subsequent ones.
- Language Expansion: The current work is specific to Mandarin. The authors suggest applying this unified structure to other languages in the future.
- Better Embeddings: The authors plan to replace the standard Word2Vec embeddings with BERT, which could provide even deeper semantic context for better prosody and pronunciation.
Summary updated