StyleTTS: A Style-Based Generative Model for Natural and Diverse Text-to-Speech Synthesis

Yinghao Aaron Li, Cong Han, Nima Mesgarani

TL;DR

StyleTTS is a novel parallel text-to-speech (TTS) model that synthesizes natural and expressive speech by leveraging style information from reference audio. It introduces a Transferable Monotonic Aligner (TMA) and duration-invariant data augmentation to improve alignment and prosody, achieving state-of-the-art results in naturalness and speaker similarity across various datasets.

What Problem Does This Paper Address?

While recent advancements in deep learning have significantly improved text-to-speech (TTS) synthesis, generating speech with naturalistic prosodic variations, speaking styles, and emotional tones remains a significant challenge. Existing parallel TTS models, which offer faster synthesis than autoregressive models, often struggle with accurately aligning text to speech and capturing the nuances of human expression. This can lead to robotic-sounding speech or an inability to replicate the subtle stylistic elements present in a reference utterance. The paper aims to bridge this gap by developing a TTS system that can synthesize diverse and expressive speech, mimicking the style of a given reference audio.

Methods

StyleTTS is a parallel TTS model built upon a novel framework with several key innovations:

  • Style-Based Generative Modeling: The core of StyleTTS is its ability to capture and utilize "style" from a reference audio utterance. This style encompasses prosodic patterns, speaking rate, emotional tone, and speaker identity, all learned without explicit labels.
  • Adaptive Instance Normalization (AdaIN): Style information is integrated into the decoder using AdaIN, a technique proven effective in image style transfer. AdaIN allows the model to adapt the generated speech's acoustic features based on the style vector derived from the reference audio. This is a key differentiator from methods that simply concatenate style embeddings.
  • Transferable Monotonic Aligner (TMA): To address the challenge of monotonic alignment in parallel TTS, StyleTTS introduces TMA. This involves fine-tuning a pre-trained text aligner (from ASR tasks) on TTS data. This approach leverages existing alignment knowledge while adapting it to the specific needs of TTS, mitigating issues with external aligners and overfitting.
  • Duration-Invariant Data Augmentation: To improve prosody prediction robustness, a novel data augmentation scheme is employed. This method stretches or compresses mel-spectrograms in time, altering speech speed while preserving pitch and energy, thus training the model to be invariant to duration variations.
  • Modular Architecture: The framework comprises several modules, including a text encoder, style encoder, pitch extractor, duration predictor, prosody predictor, and a decoder. These modules work together to generate the final mel-spectrogram.

Datasets

StyleTTS was trained and evaluated on the following datasets:

  • LJSpeech: A single-speaker dataset used for training single-speaker models.
  • LibriTTS: A multi-speaker dataset used for training multi-speaker models. The "train-clean-460" subset was utilized.
  • Emotional Speech Dataset (ESD): Used to demonstrate the model's capacity for synthesizing speech with diverse prosodic patterns and emotional tones.
  • VCTK: Used to evaluate the model's zero-shot speaker adaptation capabilities.

Key Insights and Ideas

  • AdaIN for Speech Style: The paper demonstrates that AdaIN, a technique successful in image style transfer, is highly effective for capturing and transferring complex acoustic styles in speech synthesis, outperforming simpler concatenation methods.
  • TMA for Robust Alignment: Fine-tuning pre-trained text aligners (TMA) is crucial for achieving high-quality TTS. It balances the benefits of pre-trained knowledge with the specific requirements of TTS, avoiding issues associated with purely external or purely internal aligners.
  • Self-Supervised Style Learning: StyleTTS can learn meaningful style representations, including emotional tones and speaker identities, without requiring explicit labels for these attributes. This is shown through t-SNE visualizations of style vectors.
  • Duration Invariance for Prosody: Augmenting data to be invariant to duration variations significantly improves the robustness of prosody prediction, leading to more natural speech even when duration predictions are suboptimal.

Key Results

  • Superior Naturalness and Similarity: StyleTTS consistently outperformed baseline models (Tacotron 2, FastSpeech 2, VITS) in subjective evaluations (MOS-N and MOS-S) on LJSpeech and LibriTTS datasets, achieving higher scores for both naturalness and speaker similarity.
  • Robustness to Long Utterances: StyleTTS showed improved robustness on longer text inputs, as indicated by lower Word Error Rates (WER) compared to other models on the LibriTTS dataset.
  • Zero-Shot Speaker Adaptation: The model demonstrated strong performance in zero-shot speaker adaptation on the VCTK dataset, outperforming other models like YourTTS and StyleSpeech.
  • Emotional Style Emulation: The model successfully synthesized speech that emulated the emotional and prosodic tones of reference audios, as evidenced by acoustic feature correlations.
  • Efficiency: StyleTTS achieved a faster Real-Time Factor (RTF) than VITS, despite not being trained end-to-end like VITS.

Limitations and Future Work

  • Evaluation Method Discrepancy: The authors note that their MOS evaluation results differ from some reported baseline results, attributing this to differences in evaluation methodology (e.g., MUSHRA-like evaluation with references vs. traditional MOS).
  • Potential for Overfitting: While TMA mitigates overfitting, the paper acknowledges that using 0% hard alignment (100% soft attention) can lead to overfitting and degraded performance.
  • Future Applications: The authors suggest that StyleTTS can empower various applications such as movie dubbing, book narration, unsupervised speech emotion recognition, and personalized speech generation. They also point to follow-up work on voice conversion.

Summary updated

Abstract

Text-to-Speech (TTS) has recently seen great progress in synthesizing high-quality speech owing to the rapid development of parallel TTS systems, but producing speech with naturalistic prosodic variations, speaking styles and emotional tones remains challenging. Moreover, since duration and speech are generated separately, parallel TTS models still have problems finding the best monotonic alignments that are crucial for naturalistic speech synthesis. Here, we propose StyleTTS, a style-based generative model for parallel TTS that can synthesize diverse speech with natural prosody from a reference speech utterance. With novel Transferable Monotonic Aligner (TMA) and duration-invariant data augmentation schemes, our method significantly outperforms state-of-the-art models on both single and multi-speaker datasets in subjective tests of speech naturalness and speaker similarity. Through self-supervised learning of the speaking styles, our model can synthesize speech with the same prosodic and emotional tone as any given reference speech without the need for explicitly labeling these categories.
BibTeX
@article{li2022styletts,
  author = {Yinghao Aaron Li and Cong Han and Nima Mesgarani},
  title = {{StyleTTS: A Style-Based Generative Model for Natural and Diverse Text-to-Speech Synthesis}},
  journal = {IEEE J. Sel. Top. Signal Process.},
  year = {2022},
  url = {https://openreview.net/forum?id=S2YWz_KGFB},
}
Figure