Emotional Text-To-Speech Based on Mutual-Information-Guided Emotion-Timbre Disentanglement

Jianing Yang, Sheng Li, Takahiro Shinozaki, Yuki Saito, Hiroshi Saruwatari

TL;DR

Researchers have developed a new emotional Text-To-Speech (TTS) system that captures nuanced, phoneme-level emotional expressions while keeping the speaker's unique voice (timbre) distinct. By using Mutual Information Neural Estimation (MINE) to "push apart" emotion and timbre features during training, the model achieves higher naturalness and better emotional accuracy than current state-of-the-art baselines.

What Problem Does This Paper Address?

While modern TTS systems can generate high-quality speech, making that speech sound truly emotional—especially in "zero-shot" scenarios where you only have a few seconds of a reference voice—remains difficult.

Most existing methods compress a reference audio clip into a single "global style vector." This approach has two major flaws:

  1. Loss of Detail: A single vector for an entire sentence can't capture how emotion shifts across individual sounds (phonemes).
  2. Feature Entanglement: The "style" vector often accidentally mixes the speaker's identity (timbre) with their emotion. This makes it hard to apply a "happy" emotion from one person to the voice of another without the voice sounding distorted or inconsistent.

Methods

The authors build their system on the FastSpeech 2 backbone and introduce a sophisticated Style Encoder designed to solve the entanglement problem.

  • Dual-Extractor Architecture: The Style Encoder contains two parallel paths. A Timbre Extractor captures global speaker characteristics using Global Style Tokens (GST). Simultaneously, an Emotion Extractor uses a cross-attention mechanism to align reference acoustics with specific phonemes, creating a sequence of fine-grained emotion embeddings.
  • PEPA (Phoneme-Emotion Projection Adapter): To help the model understand how text relates to emotion, this lightweight module projects linguistic features into the "prosody-rich" acoustic space learned from the reference audio.
  • MINE-Guided Disentanglement: This is the "secret sauce." The researchers used Mutual Information Neural Estimation (MINE) to calculate the dependency between the timbre and emotion features. By minimizing this mutual information, the model is forced to ensure that the timbre embedding contains only speaker info and the emotion embedding contains only prosodic info.
  • Two-Stage Training: First, the model learns basic speech synthesis using only neutral speech. In the second stage, the Style Encoder is introduced and trained with MINE and explicit emotion/speaker classifiers to provide clear optimization targets.

Datasets

The study utilized the ESD (Emotional Speech Dataset), specifically focusing on the English subset. This dataset includes 350 parallel utterances spoken by 10 native speakers across five emotions: neutral, happy, angry, sad, and surprise. The data was split 80/10/10 for training, validation, and testing.

Key Insights and Ideas

  • Phonemes Matter for Emotion: Emotion isn't a flat "filter" over a sentence; it's dynamic. By modeling emotion at the phoneme level, the system captures subtle timing and pitch inflections that global vectors miss.
  • MINE Needs a Map: The authors found that simply minimizing mutual information isn't enough because the model doesn't know which features to put where. By adding explicit Emotion and Speaker Predictors, they gave the disentanglement process a clear "direction," significantly improving results.
  • Neutral Pre-training is Critical: Starting with a "clean" encoder trained only on neutral speech prevents the linguistic model from getting "polluted" by emotional variance early on, leading to more robust synthesis later.

Key Results

The proposed model was compared against several strong baselines, including GST, StyleSpeech, MIST, and DC Comix TTS.

  • Subjective Quality: In Mean Opinion Score (MOS) tests for naturalness, the proposed model scored 3.63, outperforming all baselines. In Similarity MOS (SMOS), it scored 3.41, showing it was significantly better at mimicking the reference style.
  • Objective Accuracy: The model achieved an Unweighted Average Accuracy (UAA) of 82.42% in an emotion recognition task, vastly higher than the DC Comix TTS (58.79%) and GST (74.22%) baselines.
  • Visual Proof: t-SNE visualizations of the emotion embeddings showed tight, well-separated clusters for the five emotions. In contrast, baseline embeddings were scattered and overlapped, proving that the MINE-guided approach successfully disentangled the features.

Limitations and Future Work

The authors acknowledge a few areas for improvement:

  • Backbone Limitations: The current system uses FastSpeech 2. While reliable, the researchers plan to port this disentanglement logic to more modern backbones like diffusion models or large language model (LLM)-based TTS.
  • Modality Expansion: Future work aims to extend these fine-grained embeddings to multimodal settings (e.g., matching speech emotion to facial expressions in avatars).
  • Complexity: The two-stage training and MINE estimation add computational overhead compared to simpler global-vector methods.

Summary updated

Abstract

Current emotional Text-To-Speech (TTS) and style transfer methods rely on reference encoders to control global style or emotion vectors, but do not capture nuanced acoustic details of the reference speech. To this end, we propose a novel emotional TTS method that enables fine-grained phoneme-level emotion embedding prediction while disentangling intrinsic attributes of the reference speech. The proposed method employs a style disentanglement method to guide two feature extractors, reducing mutual information between timbre and emotion features, and effectively separating distinct style components from the reference speech. Experimental results demonstrate that our method outperforms baseline TTS systems in generating natural and emotionally rich speech. This work highlights the potential of disentangled and fine-grained representations in advancing the quality and flexibility of emotional TTS systems. 11The synthesized audio samples are available at https://baleyang.github.io/emotion-timbre-disentangled-tts/
BibTeX
@inproceedings{yang2025emotional,
  author = {Jianing Yang and Sheng Li and Takahiro Shinozaki and Yuki Saito and Hiroshi Saruwatari},
  title = {{Emotional Text-To-Speech Based on Mutual-Information-Guided Emotion-Timbre Disentanglement}},
  booktitle = {APSIPA},
  year = {2025},
  url = {https://openreview.net/forum?id=CzzsVV5NFd},
}

Loading related content...

Figure