Abstract
Existing emotional speech synthesis methods often utilize an utterance-level style embedding extracted from reference audio, neglecting the inherent multi-scale property of speech prosody. We introduce ED-TTS, a multi-scale emotional speech synthesis model that leverages Speech Emotion Diarization (SED) and Speech Emotion Recognition (SER) to model emotions at different levels. Specifically, our proposed approach integrates the utterance-level emotion embedding extracted by SER with fine-grained frame-level emotion embedding obtained from SED. These embeddings are used to condition the reverse process of the denoising diffusion probabilistic model (DDPM). Additionally, we employ cross-domain SED to accurately predict soft labels, addressing the challenge of a scarcity of fine-grained emotion-annotated datasets for supervising emotional TTS training.
Introduction
Traditional emotional text-to-speech (TTS) systems typically treat emotion as a global attribute, applying a single style (e.g., "happy" or "sad") to an entire utterance. However, human speech is naturally dynamic, often featuring transitions between neutral and intense emotions or stressing specific segments within a single sentence. Modeling these fine-grained variations is challenging due to two primary factors: the scarcity of large-scale TTS datasets with frame-level emotional annotations and the domain gap between existing Speech Emotion Diarization (SED) models and high-quality TTS audio.
Methods
ED-TTS is built upon GradTTS, a score-based diffusion model, and introduces several innovations to capture multi-scale emotional dynamics.
Multi-Scale Style Encoder
The model employs a dual-encoder approach to capture emotion at different granularities. A pre-trained Speech Emotion Recognition (SER) model extracts global utterance-level features, while a Speech Emotion Diarization (SED) model provides fine-grained frame-level features.
Content-Style Alignment
To ensure that frame-level emotional features are correctly mapped to the corresponding text, a multi-head attention block is used. This block aligns the variable-length prosodic features from the reference audio with the phoneme representations generated by the text encoder.
Cross-Domain SED with MLMMD
To overcome the lack of labeled data, the authors utilize a cross-domain transfer learning approach. An SED model is fine-tuned using Multi-layer Local Maximum Mean Discrepancy (MLMMD), which minimizes the statistical distribution gap between the source SED training data and the target TTS data across multiple neural network layers. This allows the model to generate "soft labels" (probabilities) for unlabeled TTS data, providing the necessary supervision for training.
Experimental Setup
The framework was trained and evaluated using several datasets:
- Pre-training: Emotional corpora such as IEMOCAP, RAVDESS, Emov-DB, ESD, and JL CORPUS were used for SER/SED components.
- Cross-Domain Evaluation: The Zaion Emotion Dataset (ZED), containing emotional YouTube clips, served as a testbed for diarization transfer.
- TTS Training: The BC2013-English audiobook dataset, a 70-hour expressive single-speaker dataset lacking emotional labels, was used as the target for the cross-domain approach.
Results
The performance of ED-TTS was compared against baselines such as EmoMix and FG-TTS.
Diarization Accuracy
The proposed SED-MLMMD model achieved an Emotion Diarization Error Rate (EDER) of 27.8%, outperforming the base SED model's 31.3%. This confirms the effectiveness of multi-layer adaptation in bridging the domain gap.
Synthesis Quality and Emotion Accuracy
Subjective listening tests showed that ED-TTS achieved a Similarity MOS (SMOS) of 4.10, surpassing EmoMix (4.02). Furthermore, in objective tests, the Emotion Reclassification Accuracy (ERA) for ED-TTS was 0.749, significantly higher than EmoMix's 0.623, indicating superior emotional expressiveness.
Conclusion
ED-TTS demonstrates that viewing emotions as time-varying events rather than static global attributes leads to more realistic and expressive speech synthesis. By leveraging cross-domain SED and soft-label supervision, the model successfully learns fine-grained emotional boundaries even from unlabeled datasets. Future work will focus on extending this framework to multi-speaker scenarios and exploring text-driven prediction of emotional trajectories to remove the dependency on reference audio during inference.
Summary updated