Expressive TTS Training With Frame and Style Reconstruction Loss

Rui Liu, Berrak Sisman, Guanglai Gao, Haizhou Li

TL;DR

Researchers have developed Tacotron-PL, a novel training strategy that significantly boosts the expressiveness of text-to-speech (TTS) systems. By adding a "perceptual" style reconstruction loss derived from a pre-trained emotion recognition model, the system learns to capture complex human prosody during training without requiring any extra reference signals or manual style selection at runtime.

What Problem Does This Paper Address?

Standard neural TTS systems, like Tacotron 2, are excellent at generating clear speech but often struggle with prosody—the rhythm, stress, and intonation that make speech sound human and expressive.

The core challenge is that traditional training focuses on frame-level reconstruction loss. This is like trying to recreate a painting by matching individual pixels; you might get the colors right, but you miss the "vibe" or the broader brushstrokes of style. While "Global Style Tokens" (GST) were invented to fix this, they usually require a reference audio clip or manual input at runtime to tell the model how to speak. The authors wanted a system that learns these stylistic associations implicitly from text, producing expressive speech automatically.

Methods

The authors introduce Tacotron-PL (where PL stands for Perceptual Loss). Their approach works in three stages:

  1. Stage I: The Style Descriptor: They train a Speech Emotion Recognition (SER) model on emotional speech data. This model acts as a "style expert." Because emotion and prosody are deeply linked, the internal layers of this model contain "deep style features" that represent high-level prosodic abstractions.
  2. Stage II: Dual-Loss Training: When training the TTS model, they use two objective functions:
  3. Frame Reconstruction Loss: The standard method of matching synthesized spectral features to the target, frame by frame.
  4. Style Reconstruction Loss: A novel "perceptual loss" that compares the deep style features (from the SER model) of the synthesized speech against the target speech. This forces the model to care about the "global" style of the entire utterance.
  5. Stage III: Inference: At runtime, the extra SER module is discarded. The TTS model has already "baked in" the stylistic knowledge, so it generates expressive speech directly from text just like a standard Tacotron model.

Datasets

  • IEMOCAP: Used to train the SER-based style descriptor. It contains roughly 10,000 utterances across four emotional categories (happy, angry, sad, neutral).
  • LJ-Speech: Used for the main TTS training and evaluation. This is a standard single-speaker dataset consisting of 24 hours of non-fiction book readings.

Key Insights and Ideas

  • The "Aha" Moment for Features: The researchers tested different layers of the SER model (low, middle, and high-level features). They found that low-level features ($\Psi_{low}$) were the most effective. While high-level features are great for categorizing an emotion (e.g., "This person is angry"), low-level features capture the nuanced acoustic textures and temporal patterns that define prosody.
  • Perceptual Loss for Audio: Borrowing a concept from computer vision (where perceptual loss helps generate realistic textures in images), the authors successfully applied it to the "texture" of speech.
  • Implicit Association: By using this loss during training, the model learns to associate specific text patterns with appropriate prosodic styles automatically, removing the need for "style control" knobs during actual use.

Key Results

The proposed Tacotron-PL(L) model (using low-level features) outperformed both the baseline Tacotron and the popular GST-Tacotron paradigm in several metrics:

  • Objective Metrics: It achieved the lowest Mel-cepstral distortion (MCD) at 6.37 dB (vs. 7.01 for baseline) and the best F0 modeling (RMSE) at 0.94 Hz, indicating much more accurate pitch contours.
  • Subjective Naturalness: In "Mean Opinion Score" (MOS) tests, listeners rated the Tacotron-PL speech significantly higher than the baseline (4.19 vs 3.86 when using a WaveRNN vocoder).
  • Preference: In head-to-head AB preference tests, listeners preferred the expressiveness of Tacotron-PL over the baseline more than 54% of the time.

Limitations and Future Work

  • Single Speaker Focus: The current study was conducted using the LJ-Speech dataset, which features only one speaker. The authors note that testing this on multi-speaker datasets would be a logical next step.
  • Architecture Agnostic: While this paper focused on Tacotron, the authors suggest that this "frame + style" loss strategy could be applied to other end-to-end TTS architectures (like FastSpeech or Glow-TTS).
  • SER Dependency: The quality of the expressiveness is inherently tied to the quality and diversity of the SER model used as the style descriptor. Expanding the SER training data could further refine the results.

Summary updated

Abstract

We propose a novel training strategy for Tacotron-based text-to-speech (TTS) system that improves the speech styling at utterance level. One of the key challenges in prosody modeling is the lack of reference that makes explicit modeling difficult. The proposed technique doesn't require prosody annotations from training data. It doesn't attempt to model prosody explicitly either, but rather encodes the association between input text and its prosody styles using a Tacotron-based TTS framework. This study marks a departure from the style token paradigm where prosody is explicitly modeled by a bank of prosody embeddings. It adopts a combination of two objective functions: 1) frame level reconstruction loss, that is calculated between the synthesized and target spectral features; 2) utterance level style reconstruction loss, that is calculated between the deep style features of synthesized and target speech. The style reconstruction loss is formulated as a perceptual loss to ensure that utterance level speech style is taken into consideration during training. Experiments show that the proposed training strategy achieves remarkable performance and outperforms the state-of-the-art baseline in both naturalness and expressiveness. To our best knowledge, this is the first study to incorporate utterance level perceptual quality as a loss function into Tacotron training for improved expressiveness.
BibTeX
@article{liu2021expressive,
  author = {Rui Liu and Berrak Sisman and Guanglai Gao and Haizhou Li},
  title = {{Expressive TTS Training With Frame and Style Reconstruction Loss}},
  journal = {IEEE ACM Trans. Audio Speech Lang. Process.},
  year = {2021},
  url = {https://openreview.net/forum?id=99D9mIli2t},
}
Figure