Abstract
EE-TTS (Emphatic Expressive TTS) is a novel text-to-speech framework designed to enhance speech expressiveness by automatically predicting emphasis locations based on a sentence's linguistic structure. By integrating multi-level linguistic features—including syntax and semantics—with a Gated Graph Neural Network (GGN) and a Conformer-based acoustic model, EE-TTS achieves significant improvements in naturalness and expressiveness. The system utilizes unsupervised pre-training to mitigate the need for extensive manually labeled data, demonstrating robust performance across diverse styles such as gaming and live streaming.
Introduction
While modern Text-to-Speech (TTS) systems produce high-quality, clear audio, they often sound "flat" or "robotic" in high-stakes expressive scenarios like video game dubbing or live streaming. A major reason for this is the lack of appropriate emphasis. Emphasis is not merely a change in volume; it involves complex variations in pitch, duration, and spectral energy.
Previous attempts to add emphasis often relied on manual labels—which are expensive and subjective—or simple acoustic features that failed to capture the underlying reason for a word's stress. The authors of EE-TTS argue that emphasis is deeply rooted in the underlying linguistic structure: the syntax (how words relate) and semantics (what words mean). By leveraging these features, EE-TTS bridges the gap between raw text and expressive speech without requiring exhaustive manual labeling for every input.
Methodology
The EE-TTS architecture consists of three primary modules designed to extract, predict, and synthesize expressive speech:
- Linguistic Information Extractor: This module pulls multi-level data from the text. It utilizes
jiebafor word segmentation,pyltpfor Part-of-Speech (POS) tags and Dependency Parsing (DP) to map word relationships, and a pre-trained BERT model to capture deep semantic meaning at the character level. - Emphasis Predictor: Rather than using a simple linear classifier, the researchers employed a Gated Graph Neural Network (GGN). This treats the sentence as a graph where words are nodes and dependency relations are edges, allowing the model to understand the structural importance of words when predicting emphasis positions.
- Conditioned Acoustic Model: Based on the FastSpeech2 architecture, this model replaces the standard Transformer encoder with a Conformer. Conformers are superior at modeling both local and global dependencies, which is crucial for handling the relative positioning and prosodic influence of emphasized words.
Unsupervised Pre-training
To solve the problem of data scarcity, the framework employs a "pseudo-emphasis" strategy. Using a signal-based tool (Wavelet Prosody Toolkit), the authors extracted emphasis labels from 80 hours of unlabeled Mandarin speech. This allowed the model to learn general prosodic patterns before being fine-tuned on smaller, high-quality expressive datasets.
Experimental Setup
The study focused on Mandarin Chinese, utilizing a large 80-hour pre-training set (~90,000 utterances) and two specialized fine-tuning sets:
- F1 (Gaming Style): 3,500 utterances from a female speaker with ~1,800 emphasized sentences.
- F2 (Live Streaming Style): 3,000 utterances from a female speaker.
Professional annotators provided ground-truth emphasis labels for the fine-tuning sets to evaluate the predictor's accuracy against human judgment.
Key Results and Insights
EE-TTS was evaluated using Mean Opinion Scores (MOS) on a scale of 1 to 5, showing substantial improvements over baseline models:
- Expressiveness (E-MOS): EE-TTS achieved a score of 4.24, compared to the baseline's 3.76.
- Naturalness (N-MOS): EE-TTS scored 4.37, compared to the baseline's 3.67.
The Importance of Syntax and Semantics
Ablation studies confirmed that semantic information from BERT was the most impactful feature. Removing BERT caused the F-score of emphasis prediction to drop from 0.57 to 0.43. Additionally, Dependency Parsing (DP) proved vital; by understanding if a word is a "root" or a "direct object," the model can more accurately determine its prosodic weight.
"Reasonable" vs. "Exact" Emphasis
A fascinating finding was that the model's predicted emphasis sometimes resulted in higher naturalness scores than using human-labeled ground truth. This suggests that in speech, there is rarely a single "correct" way to emphasize a sentence. As long as the placement is linguistically "reasonable," listeners perceive the output as natural and expressive.
Conclusion and Future Work
EE-TTS demonstrates that integrating structural linguistic information is key to moving beyond robotic TTS. The use of Conformers and GGNs allows the model to respect the complex relationship between a word's position in a sentence and its acoustic realization.
While the current study is limited to Mandarin and still benefits from small amounts of labeled data for fine-tuning, it provides a robust framework for future cross-lingual applications. Future work will focus on further reducing the dependency on manual labels and exploring the model's generalization to other languages and more diverse speaking styles.
Summary updated