Abstract
Current neural text-to-speech (TTS) models generate high-quality speech but struggle with fine-grained control over emotion intensity. Existing methods often require external optimizations that degrade audio quality. We present EmoDiff, a diffusion-based TTS model that utilizes a novel "soft-label guidance" technique. By guiding the denoising process with a weighted mixture of target emotion and neutral labels, EmoDiff allows for precise, linear control over emotion intensity (from 0 to 1) while maintaining high voice quality and generating diverse expressive outputs.
Introduction
While modern neural TTS models produce remarkably human-like voices, making them "act" with specific emotional intensities remains a hurdle. Most existing systems treat emotions as binary categories (e.g., "Happy" vs. "Neutral"). Previous attempts to achieve "slightly happy" voices relied on complex multi-stage optimizations like Relative Attributes Rank (RAR) or manual embedding manipulation. These methods often introduce bias or hand-crafted constraints that hurt the naturalness and audio quality of the generated speech. EmoDiff addresses this by providing a way to "turn the dial" on emotion intensity without compromising the underlying acoustic model.
Methods
EmoDiff builds upon the architecture of Grad-TTS, a diffusion-based model that generates speech by gradually denoising a signal. The framework operates in three distinct phases:
- Unconditional Acoustic Model Training: A diffusion model is trained on emotional data without explicit emotion labels, allowing it to learn the broad distribution of human speech.
- Emotion Classifier Training: A separate emotion classifier is trained on the noisy intermediate states ($x_t$) of the diffusion process, learning to identify emotions even in high-noise regimes.
- Soft-Label Guidance: During the generation (sampling) phase, instead of a "one-hot" target, a soft label is provided. The target is defined as $d = \alpha \cdot \text{Emotion} + (1 - \alpha) \cdot \text{Neutral}$, where $\alpha$ (0 to 1) acts as the intensity knob. The gradient of the classifier nudges the denoising process toward this specific weighted mixture.
Experimental Setup
The researchers utilized the English portion of the Emotional Speech Dataset (ESD), featuring 10 speakers with 350 parallel utterances per speaker per emotion (Angry, Happy, Sad, Surprise, and Neutral). Mel-spectrograms were extracted at a 12.5ms frame shift, and HifiGAN served as the vocoder for final audio synthesis.
Key Results and Insights
EmoDiff was compared against MixedEmotion (an autoregressive baseline) and a hard-conditioned Grad-TTS.
- Audio Quality: In Mean Opinion Score (MOS) tests, EmoDiff achieved a score of 4.13, significantly outperforming MixedEmotion (3.43) and matching the quality of hard-labeled Grad-TTS (4.16).
- Linear Controllability: Experiments demonstrated a near-linear correlation between the input $\alpha$ value and the resulting classification probability across various emotions. This confirms that intensity can be mathematically modeled as the "force" applied by a classifier during sampling.
- Diversity: Listeners preferred EmoDiff over MixedEmotion for its ability to produce varied and natural prosody, a benefit of the stochastic nature of diffusion models.
Discussion and Future Work
The core innovation of EmoDiff is treating emotion intensity as a probabilistic weight during sampling rather than a fixed architectural attribute. This preserves high acoustic quality because guidance happens during inference.
While successful, future work includes scaling to multi-speaker environments and exploring complex emotion mixing (e.g., combining Sad and Angry). Additionally, refining duration modeling for emotions like "Sad," where speech rate is a key intensity marker, remains an area for improvement.
Summary updated