ED-TTS: Multi-Scale Emotion Modeling Using Cross-Domain Emotion Diarization for Emotional Speech Synthesis

Haobin Tang, Xulong Zhang, Ning Cheng, Jing Xiao, Jianzong Wang

TL;DR

ED-TTS is a novel emotional text-to-speech (TTS) framework that moves beyond simple sentence-level labels to capture how emotions fluctuate within a single sentence. By leveraging Speech Emotion Diarization (SED) and cross-domain transfer learning, the model learns to synthesize speech with fine-grained emotional boundaries, even when trained on datasets that lack manual emotional annotations.

What Problem Does This Paper Address?

Most emotional TTS systems treat emotion as a global attribute—you pick "happy" or "sad," and the model applies that style to the entire utterance. However, human speech is rarely that uniform. We often stress specific segments or transition between neutral and intense emotions within a single breath.

The challenge in modeling these "fine-grained" variations is two-fold:

  1. Data Scarcity: There are very few large-scale TTS datasets that have frame-by-frame emotional labels.
  2. Domain Gap: While we have models that can perform Speech Emotion Diarization (identifying "which emotion appears when"), these models are usually trained on specific datasets that sound very different from the high-quality audio used for TTS. Simply applying a pre-trained diarization model to TTS data often results in poor accuracy due to this "distribution shift."

Methods

ED-TTS is built upon GradTTS, a score-based diffusion model. The authors introduce several key innovations to handle multi-scale emotion:

  • Multi-Scale Style Encoder: Instead of a single embedding, the model uses a dual approach. It employs a pre-trained Speech Emotion Recognition (SER) model for global utterance-level features and a Speech Emotion Diarization (SED) model for frame-level features.
  • Content-Style Alignment: To ensure the frame-level emotional "energy" matches the right words, the authors use a multi-head attention block. This aligns the variable-length prosodic features from the reference audio with the phoneme representations of the text.
  • Cross-Domain SED with MLMMD: To label unlabeled TTS data, the authors "fine-tune" an SED model using Multi-layer Local Maximum Mean Discrepancy (MLMMD). This technique reduces the statistical gap between the SED training data (source) and the TTS data (target) across multiple layers of the neural network. This allows the model to predict "soft labels" for the TTS data, which then serve as supervision during training.

Datasets

The researchers utilized a variety of datasets to train different components:

  • SER/SED Pre-training: A mix of well-known emotional corpora including IEMOCAP, RAVDESS, Emov-DB, ESD, and JL CORPUS.
  • Cross-Domain Evaluation: The Zaion Emotion Dataset (ZED), which contains emotional clips from YouTube, was used to test how well the diarization transferred to new domains.
  • TTS Training: The BC2013-English audiobook dataset. This is a 70-hour dataset of a single female speaker. Crucially, it is expressive but lacks emotional annotations, making it the perfect "unlabeled" target for their cross-domain approach.

Key Insights and Ideas

  • Emotions as "Events": The core philosophy here is that emotions should be viewed as varying speech events with clear temporal boundaries, rather than a static characteristic of the whole speech.
  • Soft Labels as Supervision: By using "soft" labels (probabilities) from the cross-domain SED model rather than "hard" categories, the TTS model can learn the nuances and uncertainties of emotional transitions.
  • The Power of Multi-Layer Adaptation: The authors found that adapting the model across multiple CNN layers (MLMMD) was significantly more effective than just adapting the final bottleneck layer. This creates a more robust "shared feature space" between the diarization model and the TTS audio.

Key Results

The authors evaluated ED-TTS against several baselines, including EmoMix and FG-TTS:

  • Diarization Accuracy: The proposed SED-MLMMD model achieved an Emotion Diarization Error Rate (EDER) of 27.8%, a significant improvement over the base SED model (31.3%).
  • Synthesis Quality: In subjective listening tests, ED-TTS achieved a Similarity MOS (SMOS) of 4.10, outperforming EmoMix (4.02).
  • Emotion Accuracy: In objective tests, the Emotion Reclassification Accuracy (ERA)—which measures if an automated tool can recognize the intended emotion in the synthesized speech—was 0.749 for ED-TTS, compared to only 0.623 for EmoMix. This suggests ED-TTS is much better at actually "landing" the intended emotion.

Limitations and Future Work

While ED-TTS shows impressive results on an expressive audiobook dataset, the paper focuses on a single-speaker scenario. Extending this to multi-speaker models where speaker identity and emotional style might be entangled is a logical next step.

Additionally, the system currently relies on reference audio to extract the multi-scale style. A future iteration might explore how to predict these fine-grained emotional "trajectories" directly from text or user-defined control curves, further reducing the need for reference speech at inference time.

Summary updated

Abstract

Existing emotional speech synthesis methods often utilize an utterance-level style embedding extracted from reference audio, neglecting the inherent multi-scale property of speech prosody. We introduce ED-TTS, a multi-scale emotional speech synthesis model that leverages Speech Emotion Diarization (SED) and Speech Emotion Recognition (SER) to model emotions at different levels. Specifically, our proposed approach integrates the utterance-level emotion embedding extracted by SER with fine-grained frame-level emotion embedding obtained from SED. These embeddings are used to condition the reverse process of the denoising diffusion probabilistic model (DDPM). Additionally, we employ cross-domain SED to accurately predict soft labels, addressing the challenge of a scarcity of fine-grained emotion-annotated datasets for supervising emotional TTS training.
BibTeX
@inproceedings{tang2024ed,
  author = {Haobin Tang and Xulong Zhang and Ning Cheng and Jing Xiao and Jianzong Wang},
  title = {{ED-TTS: Multi-Scale Emotion Modeling Using Cross-Domain Emotion Diarization for Emotional Speech Synthesis}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2024},
  url = {https://openreview.net/forum?id=yfB0SauZU8},
}

Loading related content...

Figure