QI-TTS: Questioning Intonation Control for Emotional Speech Synthesis

Haobin Tang, Xulong Zhang, Jianzong Wang, Ning Cheng, Jing Xiao

Abstract

While modern Text-to-Speech (TTS) systems have achieved high levels of naturalness in expressing global emotions, they often struggle with the nuanced prosody required for specific communicative intentions, such as questioning. This paper discusses QI-TTS, a framework designed to jointly control emotion and questioning intonation. By utilizing a multi-style extractor focused on the final syllable and a ranking-based relative attribute method, QI-TTS enables fine-grained control over intonation intensity without the need for manual intensity labels.

Introduction

In human communication, the distinction between a statement and a question is often conveyed through prosody rather than syntax alone. For instance, the phrase "You failed the exam" can be delivered as a definitive statement or a shocked inquiry, even when maintaining a consistent emotional tone like anger. Most expressive TTS models treat style as a global, sentence-level attribute, which fails to capture these local intonational shifts.

Existing emotional TTS systems face three primary challenges:

  1. Lack of Differential Prosody: Difficulty in distinguishing between statements and questions within the same emotional category.
  2. Intensity Variation: Intonation is not binary; it exists on a spectrum of strength that current models struggle to replicate.
  3. Information Overlap: The difficulty of disentangling prosodic features from linguistic content, which can degrade audio quality.

QI-TTS addresses these hurdles by isolating intonation markers and introducing a method for unsupervised intensity control.

Methodology

QI-TTS is built upon the non-autoregressive FastSpeech 2 architecture, enhanced with specialized modules for style extraction and disentanglement.

Multi-Style Extractor and Residual Embedding

Rather than extracting a single global style, the model employs a dual-style extraction strategy. It captures emotion features from the entire sentence while specifically targeting the final syllable for intonation features. Linguistic research indicates that the final syllable (approximately the last 0.52 seconds of an English utterance) contains the most critical pitch (F0) variations for signaling a question.

To isolate the intonation from the broader emotional context, the authors use a Residual Style Embedding ($R_f - R_s$). By subtracting the sentence-level embedding ($R_s$) from the final syllable embedding ($R_f$), the model isolates the specific prosodic markers associated with the questioning lilt.

Relative Attributes for Intensity Control

A significant innovation of QI-TTS is the use of a ranking-based "relative attribute" method. Because datasets typically lack granular labels for "questioning intensity," the model learns to rank audio samples based on acoustic features. It assigns a relative degree of intonation strength (ranging from 0 to 1), allowing users to adjust the intensity of the inquiry during inference.

Disentanglement via Gradient Reversal

To prevent the style embeddings from capturing linguistic information—which can lead to instability—the framework incorporates a Gradient Reversal Layer (GRL) with a content predictor. This setup forces the style extractor to ignore the text content and focus exclusively on prosodic variations.

Experimental Setup

The model was trained and evaluated using the English portion of the Emotional Speech Dataset (ESD), which includes 10 native speakers and five emotions: Neutral, Sad, Happy, Angry, and Surprise. Since the ESD does not feature explicit intonation labels, the researchers applied K-means clustering to categorize utterances into "statements" and "questions," identifying approximately 310 questions per speaker.

Results and Discussion

QI-TTS was benchmarked against baseline models such as MultiEmo FS2 and Styler using both objective metrics and subjective human listening tests.

  • Intonation Accuracy: QI-TTS achieved an accuracy of 95.2% in delivering the intended intonation (question vs. statement), significantly outperforming baselines which ranged between 81% and 85%.
  • Audio Quality: The model maintained a high Mean Opinion Score (MOS) of 3.84, indicating that the added control did not come at the cost of naturalness.
  • Intensity Perception: Best-Worst Scaling tests confirmed that listeners could consistently distinguish between different intensity settings. Higher settings (e.g., 0.9) were perceived as having a much sharper questioning quality than lower settings.

An interesting finding was the interaction between intonation and emotion perception. Increased questioning intensity was found to enhance the perception of "Surprise" but occasionally confused the perception of "Angry," suggesting a deep psychological link between specific intonations and emotional categories.

Conclusion

QI-TTS represents a significant step forward in fine-grained prosody control for TTS. By focusing on the final syllable and employing relative attribute ranking, the model provides a flexible way to synthesize speech with both emotional depth and intentional nuance. Future work will focus on expanding this framework to multilingual contexts and tonal languages, where intonation patterns differ significantly from English.

Summary updated

Abstract

Recent expressive text to speech (TTS) models focus on synthesizing emotional speech, but some fine-grained styles such as intonation are neglected. In this paper, we propose QI-TTS which aims to better transfer and control intonation to further deliver the speaker’s questioning intention while transferring emotion from reference speech. We propose a multi-style extractor to extract style embedding from two different levels. While the sentence level represents emotion, the final syllable level represents intonation. For fine-grained intonation control, we use relative attributes to represent intonation intensity at the syllable level. Experiments have validated the effectiveness of QI-TTS for improving intonation expressiveness in emotional speech synthesis.
BibTeX
@inproceedings{tang2023qi,
  author = {Haobin Tang and Xulong Zhang and Jianzong Wang and Ning Cheng and Jing Xiao},
  title = {{QI-TTS: Questioning Intonation Control for Emotional Speech Synthesis}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2023},
  url = {https://openreview.net/forum?id=frn_44ZVuZ},
}
Figure