Abstract
Current emotional Text-To-Speech (TTS) and style transfer methods rely on reference encoders to control global style or emotion vectors, but do not capture nuanced acoustic details of the reference speech. To this end, we propose a novel emotional TTS method that enables fine-grained phoneme-level emotion embedding prediction while disentangling intrinsic attributes of the reference speech. The proposed method employs a style disentanglement method to guide two feature extractors, reducing mutual information between timbre and emotion features, and effectively separating distinct style components from the reference speech. Experimental results demonstrate that our method outperforms baseline TTS systems in generating natural and emotionally rich speech. This work highlights the potential of disentangled and fine-grained representations in advancing the quality and flexibility of emotional TTS systems.
Introduction
Generating high-quality emotional speech remains a significant challenge in Text-To-Speech (TTS) research, particularly in "zero-shot" scenarios where only a brief reference audio clip is available. Traditional methods often compress reference audio into a single "global style vector." However, this approach suffers from two primary limitations: the loss of fine-grained acoustic details that vary across phonemes, and the entanglement of speaker identity (timbre) with emotional expression. This entanglement makes it difficult to transfer emotions between different speakers without introducing artifacts or losing the target speaker's unique voice characteristics.
Proposed Methodology
The proposed system is built upon the FastSpeech 2 backbone, enhanced with a sophisticated Style Encoder designed to disentangle and capture nuanced emotional features.
Dual-Extractor Architecture
The Style Encoder utilizes two parallel paths to process the reference mel-spectrogram:
- Timbre Extractor: Captures global speaker characteristics using Global Style Tokens (GST).
- Emotion Extractor: Employs a cross-attention mechanism to align reference acoustics with specific phonemes, generating a sequence of fine-grained emotion embeddings.
MINE-Guided Disentanglement
To ensure the separation of timbre and emotion, the researchers integrated Mutual Information Neural Estimation (MINE). By minimizing the mutual information between the timbre and emotion features, the model is forced to isolate speaker-specific information from prosodic emotional content. This process is further guided by explicit Emotion and Speaker Predictors, which provide clear optimization targets for the disentanglement process.
Phoneme-Emotion Projection Adapter (PEPA)
The PEPA module helps the model bridge the gap between linguistic features and the "prosody-rich" acoustic space learned from the reference audio, ensuring that the predicted emotion embeddings are contextually appropriate for the input text.
Experimental Setup
The model was evaluated using the Emotional Speech Dataset (ESD), specifically the English subset containing 350 parallel utterances from 10 native speakers across five emotions: neutral, happy, angry, sad, and surprise. The system underwent a two-stage training process, starting with neutral speech pre-training to establish a robust linguistic foundation before introducing the Style Encoder and MINE-guided disentanglement.
Results and Discussion
The proposed model was compared against several state-of-the-art baselines, including GST, StyleSpeech, MIST, and DC Comix TTS.
Quantitative Performance
As shown in the results table, the proposed method achieved the highest Mean Opinion Score (MOS: 3.63) for naturalness and Similarity MOS (SMOS: 3.41) for style mimicry. Furthermore, it reached an Unweighted Average Accuracy (UAA) of 82.42% in emotion recognition tasks, significantly outperforming baselines like DC Comix TTS (58.79%).
Qualitative Analysis
The t-SNE visualizations provide clear evidence of successful disentanglement. While baseline models show scattered and overlapping emotion embeddings, the proposed model produces tight, well-separated clusters for each of the five emotions. This confirms that the MINE-guided approach effectively captures distinct emotional categories.
Conclusion
This research demonstrates that modeling emotion at the phoneme level, combined with rigorous mutual-information-based disentanglement, significantly improves the quality and emotional accuracy of TTS systems. Future work will explore porting this disentanglement logic to modern backbones like diffusion models and extending the framework to multimodal applications, such as synchronizing speech with facial expressions.
Summary updated