Abstract
We present a novel generative model that combines state-of-the-art neural text-to-speech (TTS) with semi-supervised probabilistic latent variable models. By providing partial supervision to some of the latent variables, we are able to force them to take on consistent and interpretable purposes, which previously hasn't been possible with purely unsupervised TTS models. We demonstrate that our model is able to reliably discover and control important but rarely labelled attributes of speech, such as affect and speaking rate, with as little as 1% (30 minutes) supervision. Even at such low supervision levels we do not observe a degradation of synthesis quality compared to a state-of-the-art baseline.
Introduction
Modern neural Text-to-Speech (TTS) systems, such as Tacotron, have achieved remarkable naturalness. However, providing fine-grained control over prosodic elements like emotion, speaking rate, and pitch remains a significant challenge. Traditional supervised approaches require exhaustive labeling, which is expensive and often subjective. Conversely, unsupervised methods using Variational Autoencoders (VAEs) can discover latent styles but suffer from "non-identifiability," where latent dimensions lack consistent, interpretable meanings across training runs.
This work introduces a semi-supervised framework that "anchors" specific latent variables to human-interpretable attributes using minimal labeled data. By combining the strengths of supervised and unsupervised learning, we achieve precise control without sacrificing the scalability of unsupervised models.
Methodology
The proposed architecture augments the Tacotron sequence-to-sequence model with a VAE-based latent space. The model utilizes two distinct sets of latent variables:
- Semi-supervised variables ($z_s$): These act as explicit control knobs for attributes like affect or speaking rate. During training, these variables are supervised with labels when available.
- Unsupervised variables ($z_u$): These capture residual prosodic nuances not covered by explicit labels, ensuring the synthesized speech remains natural and varied.
Probabilistic Framework
The model is trained using a dual objective. It optimizes the standard Evidence Lower Bound (ELBO) for reconstruction while simultaneously minimizing a classification or regression loss for the supervised latent variables $z_s$. This ensures that $z_s$ is not only useful for reconstructing the audio but also aligns with the provided labels.
Crucially, the model accounts for the dependency between text ($y$) and latent variables ($z$) in the posterior, recognizing that both contribute to "explaining" the observed audio ($x$). Hyper-parameters $\gamma$ and $\alpha$ are employed to weight supervised examples and auxiliary losses, preventing posterior collapse and ensuring the model respects the provided labels even at low supervision levels.
Results and Discussion
The model was evaluated on a high-quality multi-speaker dataset (52 hours) with labels for affect, speaking rate, and F0 variation.
Data Efficiency and Control
A key finding is the "anchor effect": providing labels for as little as 1% of the training data (approximately 30 minutes) is sufficient to define interpretable directions in the latent space. Once anchored, the model generalizes these concepts across the entire unlabeled dataset. Furthermore, the learned latent space allows for extrapolation, enabling the synthesis of speech with attributes (e.g., extreme speaking rates) beyond those observed in the training set.
Quality and Transferability
Mean Opinion Score (MOS) tests indicate that the addition of control mechanisms does not degrade audio quality. The semi-supervised models achieved MOS scores (4.1-4.2) comparable to the Tacotron baseline (4.08). Additionally, the model demonstrates "controllability transfer," where attributes learned from labeled speakers can be applied to completely unlabeled speakers, such as those in the Catherine Byers audiobook dataset.
Conclusion
We have demonstrated a robust semi-supervised approach for controllable TTS. By anchoring latent variables with minimal supervision, we achieve interpretable control over speech attributes while maintaining state-of-the-art naturalness. Future work will explore more expressive output distributions and conditional priors to further refine the modeling of complex prosodic interactions.
Summary updated