Adjusting Pleasure-Arousal-Dominance for Continuous Emotional Text-to-Speech Synthesizer

Azam Rabiee, Tae-Ho Kim, Soo-Young LeeInterspeech 2019

TL;DR

This paper introduces a novel end-to-end neural text-to-speech (TTS) system capable of generating speech with continuous emotional nuances, moving beyond discrete emotion categories. By leveraging the Pleasure-Arousal-Dominance (PAD) model and a modified Tacotron architecture, they achieve more natural and controllable emotional speech synthesis.

What Problem Does This Paper Address?

Traditional text-to-speech (TTS) systems often struggle to convey the rich spectrum of human emotions. While some systems can produce discrete emotions like "happy" or "sad," human emotional expression is far more nuanced and exists on a continuous scale. Existing emotional TTS models are often limited by the availability of large, diverse emotional speech datasets and the difficulty of mapping discrete emotions to a continuous emotional space. This paper tackles the challenge of creating a TTS system that can generate speech with a virtually unlimited range of emotions by utilizing the psychologically established Pleasure-Arousal-Dominance (PAD) model.

Methods

The core of this work is an end-to-end neural speech synthesizer based on the Tacotron architecture. The key innovation lies in how emotional style is injected into the model. Instead of using discrete emotion labels, the system incorporates the 3D PAD values, representing pleasure, arousal, and dominance.

  • PAD Representation: The 3D PAD values are projected into a higher-dimensional space (32-D in their implementation) to provide richer stylistic information to the network.
  • Style Injection: The projected PAD style is injected into both the attention mechanism and the decoder modules of the Tacotron model. This allows the model to condition the speech generation on the desired emotional state.
  • PAD Adjustment: To better align the psychological PAD values with the TTS system's needs, a transfer learning approach is used. Initially, the PAD values are adapted using a pre-trained model, and then fine-tuned to improve the distinctiveness of basic emotion categories without losing the core meaning of the PAD axes.
  • Architecture Tuning: Extensive hyperparameter tuning was performed to find the optimal network architecture for style injection, considering factors like minimizing added parameters, avoiding style confusion, and preserving speech quality.

Datasets

The experiments were conducted on an internal Korean dataset. This dataset contains seven emotions (neutral and six basic emotions) uttered by a male and female speaker, with 3000 sentences per style category recorded at a 16kHz sampling rate.

Key Insights and Ideas

  • Continuous Emotion is Achievable: The paper demonstrates that it's possible to move beyond discrete emotion categories and achieve continuous emotional speech synthesis using a neural TTS model.
  • PAD as a Powerful Emotional Space: The Pleasure-Arousal-Dominance (PAD) model provides a robust and comprehensive framework for representing a wide range of human emotions, which can be effectively leveraged for TTS.
  • Higher-Dimensional Style Injection is Crucial: Simply injecting the raw 3D PAD values was insufficient. Projecting them into a higher-dimensional space significantly improved the model's ability to differentiate and convey emotional nuances.
  • Strategic Style Injection Matters: The location and type of style injection within the Tacotron architecture significantly impact performance, with attention-RNN injection proving effective without causing style confusion.
  • PAD Adjustment Enhances Distinctiveness: Fine-tuning the PAD values for the TTS task was essential for clearly distinguishing between basic emotion categories like anger, disgust, and surprise, while still preserving the underlying meaning of the PAD dimensions.

Key Results

The paper reports objective evaluation results using SDR/Mel-SDR and SD/Mel-SD metrics. The "CAT-4" model, which involves four injections into the attention-RNN and decoder, achieved the best performance. Specifically, for free-running evaluation:

  • CAT-4 Model: Achieved a Mel-SDR of 14.3 ± 0.31 dB and a Mel-SD of 6.8 ± 0.57 dB.
  • Comparison: The CAT-4 model significantly outperformed other tested models, especially in terms of Mel-SD, indicating better spectral accuracy. The authors note that without PAD adjustment, the model struggled to distinguish between certain basic emotion categories.

Limitations and Future Work

  • Limitations: The authors acknowledge that without PAD adjustment, the model had difficulty clearly distinguishing between certain basic emotion categories (angry, disgust, surprise). The paper also mentions that the "multiply" form of style injection led to exposure bias problems in free-running.
  • Future Work: The paper suggests that further exploration of network architectures and style injection strategies could lead to even better performance. The demo provided by the authors showcases the system's ability to generate speech in a wide range of emotions with continuously varying PAD axes, implying a focus on user-driven emotional control.

Summary updated

BibTeX
@inproceedings{rabiee2019adjusting,
  author = {Azam Rabiee and Tae-Ho Kim and Soo-Young Lee},
  title = {{Adjusting Pleasure-Arousal-Dominance for Continuous Emotional Text-to-Speech Synthesizer}},
  booktitle = {Interspeech},
  year = {2019},
}
Figure