Semi-Supervised Generative Modeling for Controllable Speech Synthesis

Raza Habib, Soroosh Mariooryad, Matt Shannon, Eric Battenberg, R. J. Skerry-Ryan, Daisy Stanton, David Kao, Tom Bagby

Abstract

We present a novel generative model that combines state-of-the-art neural text-to-speech (TTS) with semi-supervised probabilistic latent variable models. By providing partial supervision to some of the latent variables, we are able to force them to take on consistent and interpretable purposes, which previously hasn't been possible with purely unsupervised TTS models. We demonstrate that our model is able to reliably discover and control important but rarely labelled attributes of speech, such as affect and speaking rate, with as little as 1% (30 minutes) supervision. Even at such low supervision levels we do not observe a degradation of synthesis quality compared to a state-of-the-art baseline.

Introduction

Modern neural Text-to-Speech (TTS) systems, such as Tacotron, have achieved remarkable naturalness. However, providing fine-grained control over prosodic elements like emotion, speaking rate, and pitch remains a significant challenge. Traditional supervised approaches require exhaustive labeling, which is expensive and often subjective. Conversely, unsupervised methods using Variational Autoencoders (VAEs) can discover latent styles but suffer from "non-identifiability," where latent dimensions lack consistent, interpretable meanings across training runs.

This work introduces a semi-supervised framework that "anchors" specific latent variables to human-interpretable attributes using minimal labeled data. By combining the strengths of supervised and unsupervised learning, we achieve precise control without sacrificing the scalability of unsupervised models.

Methodology

The proposed architecture augments the Tacotron sequence-to-sequence model with a VAE-based latent space. The model utilizes two distinct sets of latent variables:

  1. Semi-supervised variables ($z_s$): These act as explicit control knobs for attributes like affect or speaking rate. During training, these variables are supervised with labels when available.
  2. Unsupervised variables ($z_u$): These capture residual prosodic nuances not covered by explicit labels, ensuring the synthesized speech remains natural and varied.
The model architecture, showing the CBHG encoder, the latent variable injection, and the attention-based decoder.

Probabilistic Framework

The model is trained using a dual objective. It optimizes the standard Evidence Lower Bound (ELBO) for reconstruction while simultaneously minimizing a classification or regression loss for the supervised latent variables $z_s$. This ensures that $z_s$ is not only useful for reconstructing the audio but also aligns with the provided labels.

Probabilistic graphical models for (a) the generative process, (b) the supervised posterior, and (c) the unsupervised posterior.

Crucially, the model accounts for the dependency between text ($y$) and latent variables ($z$) in the posterior, recognizing that both contribute to "explaining" the observed audio ($x$). Hyper-parameters $\gamma$ and $\alpha$ are employed to weight supervised examples and auxiliary losses, preventing posterior collapse and ensuring the model respects the provided labels even at low supervision levels.

Results and Discussion

The model was evaluated on a high-quality multi-speaker dataset (52 hours) with labels for affect, speaking rate, and F0 variation.

Data Efficiency and Control

A key finding is the "anchor effect": providing labels for as little as 1% of the training data (approximately 30 minutes) is sufficient to define interpretable directions in the latent space. Once anchored, the model generalizes these concepts across the entire unlabeled dataset. Furthermore, the learned latent space allows for extrapolation, enabling the synthesis of speech with attributes (e.g., extreme speaking rates) beyond those observed in the training set.

Quality and Transferability

Mean Opinion Score (MOS) tests indicate that the addition of control mechanisms does not degrade audio quality. The semi-supervised models achieved MOS scores (4.1-4.2) comparable to the Tacotron baseline (4.08). Additionally, the model demonstrates "controllability transfer," where attributes learned from labeled speakers can be applied to completely unlabeled speakers, such as those in the Catherine Byers audiobook dataset.

Conclusion

We have demonstrated a robust semi-supervised approach for controllable TTS. By anchoring latent variables with minimal supervision, we achieve interpretable control over speech attributes while maintaining state-of-the-art naturalness. Future work will explore more expressive output distributions and conditional priors to further refine the modeling of complex prosodic interactions.

Summary updated

Abstract

We present a novel generative model that combines state-of-the-art neural text-to-speech (TTS) with semi-supervised probabilistic latent variable models. By providing partial supervision to some of the latent variables, we are able to force them to take on consistent and interpretable purposes, which previously hasn't been possible with purely unsupervised TTS models. We demonstrate that our model is able to reliably discover and control important but rarely labelled attributes of speech, such as affect and speaking rate, with as little as 1% (30 minutes) supervision. Even at such low supervision levels we do not observe a degradation of synthesis quality compared to a state-of-the-art baseline. Audio samples are available on the web.
BibTeX
@inproceedings{habib2020semi,
  author = {Raza Habib and Soroosh Mariooryad and Matt Shannon and Eric Battenberg and R. J. Skerry-Ryan and Daisy Stanton and David Kao and Tom Bagby},
  title = {{Semi-Supervised Generative Modeling for Controllable Speech Synthesis}},
  booktitle = {International Conference on Learning Representations},
  year = {2020},
  url = {https://openreview.net/forum?id=9xyWIW9SyCS},
}
Figure