Semi-Supervised Generative Modeling for Controllable Speech Synthesis

Raza Habib, Soroosh Mariooryad, Matt Shannon, Eric Battenberg, R. J. Skerry-Ryan, Daisy Stanton, David Kao, Tom Bagby

Overview

Researchers from Google and UCL have developed a way to gain precise control over speech attributes—like emotion, pitch variation, and speaking rate—using a semi-supervised generative model. By labeling as little as 30 minutes of audio, the model learns to associate specific "knobs" with interpretable traits, allowing for high-quality, controllable speech synthesis without the need for massive labeled datasets.

The Challenge of Controllable Speech

In the world of Neural Text-to-Speech (TTS), we’ve become very adept at making machines sound human. However, making them sound specifically human—angry, excited, or talking quickly—remains a major challenge.

Existing unsupervised models, such as those using Variational Autoencoders (VAEs), are great at discovering "latent" features of speech, but they are non-identifiable. This means that after training, a researcher has to manually hunt through the model's internal variables to figure out which one controls "speed" and which one controls "pitch." Even worse, if you retrain the model, those variables move around. To get reliable, "hard-coded" control, you usually need massive amounts of high-quality labeled data, which is expensive and time-consuming to produce.

A Semi-Supervised Solution

The authors build upon the standard Tacotron 2 architecture but introduce a semi-supervised framework within a VAE. The core innovation lies in how they handle the latent variables—the internal "codes" the model uses to represent style.

Graphical models showing the generative process and posteriors for the semi-supervised framework.
  1. Split Latent Space: They divide the latent variables into two distinct groups: $z_u$ (fully unsupervised) and $z_s$ (semi-supervised). The $z_u$ variables capture the messy, unlabelled nuances of speech (like specific stress patterns or breathiness), while $z_s$ is forced to represent the specific attributes we want to control (like affect or speed).
  2. Modified Objective Function: During training, if a piece of audio has a label (e.g., "this is a happy voice"), the model is penalized if its internal $z_s$ doesn't match that label. If the audio is unlabelled, the model "guesses" the label and proceeds.
  3. Classification Loss: They add a specific classification/regression loss term to the training objective. This ensures that the latent space doesn't just "cluster" data, but actually aligns with human-understandable categories.
The modified Tacotron 2 architecture incorporating the split latent variables ($z_u$ and $z_s$) into the sequence-to-sequence network.

Datasets Used

To validate their approach, the researchers utilized several datasets:

  • Proprietary Google Dataset: 40 English speakers totaling 45 hours of high-quality audio.
  • Labels: Discrete labels for affect (6 emotional states based on the "circumplex model" of emotion) and continuous labels for speaking rate (syllables per second) and F0 variation (pitch range).
  • LibriTTS: A public dataset used to demonstrate that the results are reproducible outside of proprietary data.
  • Catherine Byers (Blizzard Challenge): An unlabelled audiobook dataset used to test "controllability transfer"—applying the learned controls to a completely new speaker.

Key Insights and Results

The Power of 1%

One of the most striking findings is that you don't need much data. With just 30 minutes of labeled audio (roughly 1% of the training set), the model achieves nearly the same level of control as it does with 45 hours of labels. A fully supervised model typically needs at least 5 hours of data to reach reasonable quality, highlighting the efficiency of this semi-supervised approach.

Identifiability by Design

By supervising just a few dimensions of the latent space, the model becomes "identifiable." Engineers no longer have to guess which variable does what; the "speed" knob is guaranteed to control speed.

Disentanglement and Extrapolation

The model successfully separates "what" is being said from "how" it is being said. You can hold the emotion constant while the unsupervised latents ($z_u$) still allow for natural, varied prosody across different samples. Furthermore, for continuous traits like speaking rate, the model learns the concept of speed, allowing it to generate speech that is faster or slower than anything it actually heard during training.

High Quality Maintained

Often, adding constraints to a model makes the output sound robotic. In subjective listening tests, users significantly preferred the controlled models for affect. When told to be "happy," listeners overwhelmingly identified it as such compared to a baseline. Crucially, the Mean Opinion Scores (MOS) remained high (~4.1 to 4.2), showing no degradation in naturalness compared to state-of-the-art unsupervised baselines.

Limitations and Future Work

While highly effective, the training process can be sensitive. If the "supervision weight" isn't tuned correctly, the model might ignore the labels entirely or the latent space might "collapse" into a single point. Additionally, the authors noted that the simple distributions used for the latent variables (like Gaussian or Laplace) might be too simple to capture the full complexity of human emotion.

Future directions include moving toward more expressive "priors" (the mathematical assumptions about the shape of the latent space) and exploring how to control even more subtle aspects of speech, such as sarcasm or intent.

Reference: Habib, R., et al. (2020). Semi-Supervised Generative Modeling for Controllable Speech Synthesis. Published as a conference paper at ICLR 2020.

Summary updated

Abstract

We present a novel generative model that combines state-of-the-art neural text- to-speech (TTS) with semi-supervised probabilistic latent variable models. By providing partial supervision to some of the latent variables, we are able to force them to take on consistent and interpretable purposes, which previously hasn’t been possible with purely unsupervised methods. We demonstrate that our model is able to reliably discover and control important but rarely labelled attributes of speech, such as affect and speaking rate, with as little as 1% (30 minutes) supervision. Even at such low supervision levels we do not observe a degradation of synthesis quality compared to a state-of-the-art baseline. We will release audio samples at https://google.github.io/tacotron/publications/semisupervised_generative_modeling_for_controllable_speech_synthesis/.
BibTeX
@inproceedings{habib2020semi,
  author = {Raza Habib and Soroosh Mariooryad and Matt Shannon and Eric Battenberg and R. J. Skerry-Ryan and Daisy Stanton and David Kao and Tom Bagby},
  title = {{Semi-Supervised Generative Modeling for Controllable Speech Synthesis}},
  booktitle = {International Conference on Learning Representations},
  year = {2020},
  url = {https://openreview.net/forum?id=9xyWIW9SyCS},
}
Figure