Learning latent representations for style control and transfer in end-to-end speech synthesis

Yajie Zhang, Shifeng Pan, Lei He, Zhen-Hua Ling

Abstract

In this paper, we introduce the Variational Autoencoder (VAE) to an end-to-end speech synthesis model, to learn the latent representation of speaking styles in an unsupervised manner. The style representation learned through VAE shows good properties such as disentangling, scaling, and combination, which makes it easy for style control. Style transfer can be achieved in this framework by first inferring style representation through the recognition network of VAE, then feeding it into TTS network to guide the style in synthesizing speech. To avoid Kullback-Leibler (KL) divergence collapse in training, several techniques are adopted. Finally, the proposed model shows good performance of style control and outperforms Global Style Token (GST) model in ABX preference tests on style transfer.

Introduction

Modern end-to-end text-to-speech (TTS) systems, such as Tacotron 2, are capable of generating high-quality, human-like speech. However, these models often default to a neutral speaking style. Incorporating expressive elements—such as emotion, intent, and varied prosody—typically requires large, manually labeled datasets, which are expensive and time-consuming to produce.

This research addresses the challenge of learning complex stylistic nuances in an unsupervised manner. While previous methods like Global Style Tokens (GST) utilized a bank of embeddings to capture style, they often lacked interpretability and precise control. We propose a more flexible, continuous, and disentangled representation of style using a Variational Autoencoder (VAE) integrated into the Tacotron 2 architecture. This allows for not only mimicking a reference voice but also fine-tuning specific attributes like pitch and speed.

Methodology

The proposed architecture combines the Tacotron 2 TTS backbone with a VAE-based recognition model to capture latent style variables.

Architecture of the proposed VAE-based Tacotron 2 model, highlighting the Recognition model and its integration with the TTS pipeline.

Model Components

  1. Recognition Model (Encoder): A reference audio clip is processed through a convolutional and GRU-based encoder to produce a latent variable ($z$) representing the style.
  2. Latent Space: The VAE outputs a probability distribution (defined by mean $\mu$ and variance $\sigma$). A style vector $z$ is sampled from this distribution and concatenated with the text encoder's output.
  3. TTS Decoder: The combined text and style information is passed to a location-sensitive attention network and a decoder to generate mel-spectrograms. These are then converted to audio using a WaveNet vocoder.

Training Stability

To prevent "KL divergence collapse"—where the model ignores the latent variables—two specific techniques were employed:

  • KL Annealing: The weight of the KL loss term is gradually increased during training.
  • K-step Updates: The KL loss is calculated only once every $K$ steps, preventing the latent space from collapsing prematurely.

Experimental Setup

The model was trained on the Blizzard Challenge 2013 dataset, consisting of 105 hours of audiobook recordings from a single female English speaker. Despite the single-speaker limitation, the dataset's rich "storytelling" variety provides significant prosodic diversity. Audio was preprocessed into 80-dimensional mel-spectrograms with a 12.5ms frame shift.

Results and Discussion

The VAE-based approach demonstrates significant advantages in style disentanglement and control.

Mel-spectrograms demonstrating the model's ability to manipulate speech characteristics by navigating the continuous latent space.

Style Disentanglement and Interpolation

The model successfully separated stylistic features into distinct dimensions of the latent vector $z$. For instance, specific dimensions were found to control pitch height or local pitch variation independently. Because the latent space is continuous, the model supports smooth interpolation between styles. By navigating between vectors representing different prosodic extremes (e.g., fast/high-pitch vs. slow/low-pitch), the system can generate speech at any intermediate point.

Style Transfer and Additivity

The latent representations also exhibit additivity; combining vectors for different traits (e.g., "high pitch" and "high variation") results in speech that inherits both characteristics. In ABX preference tests, human listeners consistently preferred the VAE model over the GST baseline for style transfer tasks. Notably, the model performed well in non-parallel style transfer, indicating it captures a generalized essence of style rather than simple acoustic mimicry.

Conclusion

By integrating a VAE into an end-to-end TTS framework, we achieved unsupervised discovery of interpretable style "knobs." While the current study is limited to a single speaker and some latent dimensions remain difficult to interpret, the results demonstrate a powerful path toward more expressive and controllable speech synthesis. Future work will focus on multi-speaker datasets and further refining the disentanglement of latent factors.

Summary updated

Abstract

In this paper, we introduce the Variational Autoencoder (VAE) to an end-to-end speech synthesis model, to learn the latent representation of speaking styles in an unsupervised manner. The style representation learned through VAE shows good properties such as disentangling, scaling, and combination, which makes it easy for style control. Style transfer can be achieved in this framework by first inferring style representation through the recognition network of VAE, then feeding it into TTS network to guide the style in synthesizing speech. To avoid Kullback-Leibler (KL) divergence collapse in training, several techniques are adopted. Finally, the proposed model shows good performance of style control and outperforms Global Style Token (GST) model in ABX preference tests on style transfer.
BibTeX
@inproceedings{zhang2019learning,
  author = {Yajie Zhang and Shifeng Pan and Lei He and Zhen-Hua Ling},
  title = {{Learning latent representations for style control and transfer in end-to-end speech synthesis}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2019},
  url = {https://openreview.net/forum?id=Svx9EzN9dK},
}
Figure