Learning Latent Representations for Style Control and Transfer in End-to-end Speech Synthesis

Yajie Zhang, Shifeng Pan, Lei He, Zhen-Hua Ling

TL;DR

This paper introduces a Variational Autoencoder (VAE) into the Tacotron 2 architecture to learn unsupervised, continuous latent representations of speaking styles. By mapping style to a disentangled latent space, the model enables precise style control, smooth interpolation between different prosodies, and superior style transfer from reference audio compared to existing "Global Style Token" (GST) methods.

What Problem Does This Paper Address?

While modern end-to-end text-to-speech (TTS) systems like Tacotron 2 can produce near-human quality speech, they typically default to a "neutral" speaking style. Creating "expressive" speech—incorporating emotion, intent, and prosody—usually requires expensive, manually labeled datasets.

The challenge is to learn these complex stylistic nuances in an unsupervised manner. Previous attempts, such as Global Style Tokens (GST), used a bank of embeddings to capture style, but these can be difficult to interpret or manipulate precisely. This paper seeks a more flexible, continuous, and "disentangled" way to represent style so that researchers can not only mimic a reference voice but also "dial in" specific attributes like pitch or speed.

Methods

The authors combine two powerful frameworks: Tacotron 2 (the TTS backbone) and a Variational Autoencoder (VAE).

  1. The Recognition Model (Encoder): A reference audio clip is passed through a convolutional and GRU-based encoder. This produces a "latent variable" ($z$) that represents the style.
  2. The Latent Space: Instead of a single fixed vector, the VAE outputs a probability distribution (mean and variance). The model samples from this distribution to get $z$, which is then added to the text encoder's output.
  3. The TTS Decoder: The combined text and style information is passed to a location-sensitive attention network and a decoder to produce mel-spectrograms, which are finally turned into audio via a WaveNet vocoder.
  4. Solving "KL Collapse": A common issue in VAE training is "KL divergence collapse," where the model ignores the latent variables entirely. The authors used two tricks to fix this:
  5. KL Annealing: Gradually increasing the weight of the KL loss term during training.
  6. K-step Updates: Only calculating the KL loss once every $K$ steps to keep the latent space from collapsing to zero.

Datasets

The model was trained on the Blizzard Challenge 2013 dataset.

  • Content: 105 hours of audiobook recordings.
  • Speaker: A single female English speaker.
  • Variety: Despite being one speaker, the dataset is rich in "storytelling" styles, providing the necessary prosodic diversity for the VAE to learn from.
  • Preprocessing: 80-dimensional mel-spectrograms were extracted with a 12.5ms frame shift.

Key Insights and Ideas

  • Disentanglement is Key: The VAE successfully separated different stylistic features into different dimensions of the latent vector $z$. For example, the authors found that adjusting "Dimension 6" specifically controlled pitch height, while "Dimension 10" controlled local pitch variation.
  • Style as a Spectrum: Because the latent space is continuous, you can perform interpolation. By taking the $z$ vector for "fast/high-pitch" and the $z$ vector for "slow/low-pitch," you can mathematically slide between them to generate speech at any intermediate speed or pitch.
  • Style Additivity: The latent representations are somewhat additive. If you have a vector for "high pitch" and another for "high variation," adding them together produces speech that inherits both traits.
  • Unsupervised Discovery: The model discovered these stylistic "knobs" entirely on its own, without any labels telling it what "pitch" or "speed" were.

Key Results

  • Style Transfer Performance: In ABX preference tests (where human listeners choose which audio sounds more like a reference), the VAE model consistently outperformed the baseline GST model.
  • Generalization: The model showed particularly strong performance in non-parallel style transfer (applying the style of Reference Audio A to the text of Sentence B). This suggests the VAE captures a more generalized "essence" of style rather than just mimicking the specific acoustics of the reference.
  • Stability: The use of KL annealing and K-step updates successfully prevented the "posterior collapse" that often plagues VAE-based generative models.

Limitations and Future Work

  • Single Speaker Constraint: The current study focused on a single-speaker dataset. The authors acknowledge that the next step is extending this to multi-speaker style transfer, which introduces the challenge of separating "speaker identity" from "speaking style."
  • Interpretability: While some dimensions were easily identified (like pitch), not all 32 dimensions of the latent space are immediately interpretable to humans.
  • Refining Disentanglement: Future work will focus on achieving even cleaner separation between latent factors to ensure that changing one attribute (like speed) has zero effect on others (like volume).

Summary updated

Abstract

In this paper, we introduce the Variational Autoencoder (VAE) to an end-to-end speech synthesis model, to learn the latent representation of speaking styles in an unsupervised manner. The style representation learned through VAE shows good properties such as disentangling, scaling, and combination, which makes it easy for style control. Style transfer can be achieved in this framework by first inferring style representation through the recognition network of VAE, then feeding it into TTS network to guide the style in synthesizing speech. To avoid Kullback-Leibler (KL) divergence collapse in training, several techniques are adopted. Finally, the proposed model shows good performance of style control and outperforms Global Style Token (GST) model in ABX preference tests on style transfer.
BibTeX
@inproceedings{zhang2019learning,
  author = {Yajie Zhang and Shifeng Pan and Lei He and Zhen-Hua Ling},
  title = {{Learning Latent Representations for Style Control and Transfer in End-to-end Speech Synthesis}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2019},
  url = {https://openreview.net/forum?id=Svx9EzN9dK},
}

Loading related content...

Figure