EmoDiff: Intensity Controllable Emotional Text-to-Speech with Soft-Label Guidance

Yiwei Guo, Chenpeng Du, Xie Chen, Kai Yu

Abstract

Current neural text-to-speech (TTS) models generate high-quality speech but struggle with fine-grained control over emotion intensity. Existing methods often require external optimizations that degrade audio quality. We present EmoDiff, a diffusion-based TTS model that utilizes a novel "soft-label guidance" technique. By guiding the denoising process with a weighted mixture of target emotion and neutral labels, EmoDiff allows for precise, linear control over emotion intensity (from 0 to 1) while maintaining high voice quality and generating diverse expressive outputs.

Introduction

While modern neural TTS models produce remarkably human-like voices, making them "act" with specific emotional intensities remains a hurdle. Most existing systems treat emotions as binary categories (e.g., "Happy" vs. "Neutral"). Previous attempts to achieve "slightly happy" voices relied on complex multi-stage optimizations like Relative Attributes Rank (RAR) or manual embedding manipulation. These methods often introduce bias or hand-crafted constraints that hurt the naturalness and audio quality of the generated speech. EmoDiff addresses this by providing a way to "turn the dial" on emotion intensity without compromising the underlying acoustic model.

Methods

EmoDiff builds upon the architecture of Grad-TTS, a diffusion-based model that generates speech by gradually denoising a signal. The framework operates in three distinct phases:

  1. Unconditional Acoustic Model Training: A diffusion model is trained on emotional data without explicit emotion labels, allowing it to learn the broad distribution of human speech.
  2. Emotion Classifier Training: A separate emotion classifier is trained on the noisy intermediate states ($x_t$) of the diffusion process, learning to identify emotions even in high-noise regimes.
  3. Soft-Label Guidance: During the generation (sampling) phase, instead of a "one-hot" target, a soft label is provided. The target is defined as $d = \alpha \cdot \text{Emotion} + (1 - \alpha) \cdot \text{Neutral}$, where $\alpha$ (0 to 1) acts as the intensity knob. The gradient of the classifier nudges the denoising process toward this specific weighted mixture.
Overview of the EmoDiff framework, showing the training and sampling stages.

Experimental Setup

The researchers utilized the English portion of the Emotional Speech Dataset (ESD), featuring 10 speakers with 350 parallel utterances per speaker per emotion (Angry, Happy, Sad, Surprise, and Neutral). Mel-spectrograms were extracted at a 12.5ms frame shift, and HifiGAN served as the vocoder for final audio synthesis.

Key Results and Insights

EmoDiff was compared against MixedEmotion (an autoregressive baseline) and a hard-conditioned Grad-TTS.

  • Audio Quality: In Mean Opinion Score (MOS) tests, EmoDiff achieved a score of 4.13, significantly outperforming MixedEmotion (3.43) and matching the quality of hard-labeled Grad-TTS (4.16).
  • Linear Controllability: Experiments demonstrated a near-linear correlation between the input $\alpha$ value and the resulting classification probability across various emotions. This confirms that intensity can be mathematically modeled as the "force" applied by a classifier during sampling.
  • Diversity: Listeners preferred EmoDiff over MixedEmotion for its ability to produce varied and natural prosody, a benefit of the stochastic nature of diffusion models.
Correlation between input intensity alpha and classification probability for different emotions.

Discussion and Future Work

The core innovation of EmoDiff is treating emotion intensity as a probabilistic weight during sampling rather than a fixed architectural attribute. This preserves high acoustic quality because guidance happens during inference.

While successful, future work includes scaling to multi-speaker environments and exploring complex emotion mixing (e.g., combining Sad and Angry). Additionally, refining duration modeling for emotions like "Sad," where speech rate is a key intensity marker, remains an area for improvement.

Summary updated

Abstract

Although current neural text-to-speech (TTS) models are able to generate high-quality speech, intensity controllable emotional TTS is still a challenging task. Most existing methods need external optimizations for intensity calculation, leading to suboptimal results or degraded quality. In this paper, we propose EmoDiff, a diffusion-based TTS model where emotion intensity can be manipulated by a proposed soft-label guidance technique derived from classifier guidance. Specifically, instead of being guided with a one-hot vector for the specified emotion, EmoDiff is guided with a soft label where the value of the specified emotion and \textit{Neutral} is set to $\alpha$ and $1-\alpha$ respectively. The $\alpha$ here represents the emotion intensity and can be chosen from 0 to 1. Our experiments show that EmoDiff can precisely control the emotion intensity while maintaining high voice quality. Moreover, diverse speech with specified emotion intensity can be generated by sampling in the reverse denoising process.
BibTeX
@inproceedings{guo2023emodiff,
  author = {Yiwei Guo and Chenpeng Du and Xie Chen and Kai Yu},
  title = {{EmoDiff: Intensity Controllable Emotional Text-to-Speech with Soft-Label Guidance}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2023},
  doi = {10.1109/icassp49357.2023.10095621},
}
Figure