Multi-Step Prediction and Control of Hierarchical Emotion Distribution in Text-to-Speech Synthesis

Sho Inoue, Kun Zhou, Shuai Wang, Haizhou Li

Abstract

This paper introduces a multi-step framework for predicting and controlling speech emotion across a hierarchy of levels: utterance, word, and phoneme. By leveraging global emotional context to progressively refine local variations, the proposed system achieves significantly more natural, expressive, and precisely controllable text-to-speech (TTS) synthesis compared to traditional single-step approaches.

Introduction

While modern TTS systems have achieved high levels of naturalness, they often struggle with the emotional nuances of human speech. Most existing models treat emotion as either a global style or a flat sequence of local variations. However, human emotion is inherently hierarchical: a sentence-level emotional tone is realized through specific word-level emphasis and fine-grained phoneme-level prosodic patterns (changes in pitch, duration, and energy). Failing to model this hierarchy results in speech that feels emotionally inconsistent or "flat." Furthermore, providing users with fine-grained, quantitative control over these nuances remains a significant challenge.

Hierarchical Emotion Distribution (ED)

The core of the proposed framework is the use of "soft labels" called Emotion Distributions (EDs). Instead of categorical labels (e.g., "Happy"), EDs are vectors representing the intensity of multiple emotions—Angry, Happy, Sad, and Surprise—at every level of the hierarchy. This allows for a more nuanced representation of emotional state than binary classification.

The hierarchical structure of Emotion Distributions (EDs) across utterance, word, and phoneme levels for a sample sentence.

Multi-Step Prediction and Control

Unlike previous single-step models that predict all levels simultaneously, this framework employs a multi-step approach that moves from "global to local." The model predicts the utterance-level ED first, providing a global context that informs word-level prediction, which in turn guides phoneme-level prediction. This refinement process mimics how humans modulate speech—starting with an overall tone and refining it dynamically.

The authors demonstrate that this framework can be integrated into TTS backbones either as an external, model-agnostic module or directly into a model's internal Variance Adaptor. This flexibility allows the system to support three primary operational modes: direct prediction from text, user-controlled intensity adjustment, and emotion editing of existing audio.

The proposed system supports three operational modes: (a) TTS with emotion prediction, (b) TTS with emotion control, and (c) emotion editing.

Experimental Evaluation

The researchers utilized the LibriTTS-R dataset for base TTS training and the Emotion Speech Dataset (ESD) for emotional modeling. Results from MUSHRA (listening) tests indicate that the multi-step framework significantly improves naturalness and emotional similarity compared to single-step baselines. For instance, the multi-step approach achieved emotional similarity scores of up to 62.2, compared to 57.5 for single-step models.

Key insights from the evaluation include:

  • Contextual Refinement: Local emotional variations are more coherent when anchored by global emotional intent.
  • Quantifiable Control: Subjective Best-Worst Scaling tests confirmed the system's ability to translate precise intensity adjustments (e.g., 0.0 to 1.0) into perceivable emotional changes.
  • Propagation of Emotion: Analysis suggests that emotion propagates more consistently from the utterance level to the word level than to the phoneme level, highlighting words as primary units of emotional meaning.

Conclusion and Future Work

The multi-step hierarchical approach provides a robust framework for expressive and controllable TTS. While the current study focused on English and Mandarin, future work aims to extend this to more languages and a wider variety of voice qualities. Additionally, the authors plan to address potential error accumulation in tightly coupled architectures where higher-level prediction errors can propagate to lower-level prosodic metrics.

Summary updated

Abstract

We investigate hierarchical emotion distribution (ED) for achieving multi-level quantitative control of emotion rendering in text-to-speech synthesis (TTS). We introduce a novel multi-step hierarchical ED prediction module that quantifies emotion variance at the utterance, word, and phoneme levels. By predicting emotion variance in a multi-step manner, we leverage global emotional context to refine local emotional variations, thereby capturing the intrinsic hierarchical structure of speech emotion. Our approach is validated through its integration into a variance adaptor and an external module design compatible with various TTS systems. Both objective and subjective evaluations demonstrate that the proposed framework significantly enhances emotional expressiveness and enables precise control of emotion rendering across multiple speech granularities.
BibTeX
@misc{inoue2025multi,
  author = {Sho Inoue and Kun Zhou and Shuai Wang and Haizhou Li},
  title = {{Multi-Step Prediction and Control of Hierarchical Emotion Distribution in Text-to-Speech Synthesis}},
  year = {2025},
  eprint = {2507.04598},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2507.04598},
}
Figure