Overview
Researchers from Google and UCL have developed a way to gain precise control over speech attributes—like emotion, pitch variation, and speaking rate—using a semi-supervised generative model. By labeling as little as 30 minutes of audio, the model learns to associate specific "knobs" with interpretable traits, allowing for high-quality, controllable speech synthesis without the need for massive labeled datasets.
The Challenge of Controllable Speech
In the world of Neural Text-to-Speech (TTS), we’ve become very adept at making machines sound human. However, making them sound specifically human—angry, excited, or talking quickly—remains a major challenge.
Existing unsupervised models, such as those using Variational Autoencoders (VAEs), are great at discovering "latent" features of speech, but they are non-identifiable. This means that after training, a researcher has to manually hunt through the model's internal variables to figure out which one controls "speed" and which one controls "pitch." Even worse, if you retrain the model, those variables move around. To get reliable, "hard-coded" control, you usually need massive amounts of high-quality labeled data, which is expensive and time-consuming to produce.
A Semi-Supervised Solution
The authors build upon the standard Tacotron 2 architecture but introduce a semi-supervised framework within a VAE. The core innovation lies in how they handle the latent variables—the internal "codes" the model uses to represent style.
- Split Latent Space: They divide the latent variables into two distinct groups: $z_u$ (fully unsupervised) and $z_s$ (semi-supervised). The $z_u$ variables capture the messy, unlabelled nuances of speech (like specific stress patterns or breathiness), while $z_s$ is forced to represent the specific attributes we want to control (like affect or speed).
- Modified Objective Function: During training, if a piece of audio has a label (e.g., "this is a happy voice"), the model is penalized if its internal $z_s$ doesn't match that label. If the audio is unlabelled, the model "guesses" the label and proceeds.
- Classification Loss: They add a specific classification/regression loss term to the training objective. This ensures that the latent space doesn't just "cluster" data, but actually aligns with human-understandable categories.
Datasets Used
To validate their approach, the researchers utilized several datasets:
- Proprietary Google Dataset: 40 English speakers totaling 45 hours of high-quality audio.
- Labels: Discrete labels for affect (6 emotional states based on the "circumplex model" of emotion) and continuous labels for speaking rate (syllables per second) and F0 variation (pitch range).
- LibriTTS: A public dataset used to demonstrate that the results are reproducible outside of proprietary data.
- Catherine Byers (Blizzard Challenge): An unlabelled audiobook dataset used to test "controllability transfer"—applying the learned controls to a completely new speaker.
Key Insights and Results
The Power of 1%
One of the most striking findings is that you don't need much data. With just 30 minutes of labeled audio (roughly 1% of the training set), the model achieves nearly the same level of control as it does with 45 hours of labels. A fully supervised model typically needs at least 5 hours of data to reach reasonable quality, highlighting the efficiency of this semi-supervised approach.
Identifiability by Design
By supervising just a few dimensions of the latent space, the model becomes "identifiable." Engineers no longer have to guess which variable does what; the "speed" knob is guaranteed to control speed.
Disentanglement and Extrapolation
The model successfully separates "what" is being said from "how" it is being said. You can hold the emotion constant while the unsupervised latents ($z_u$) still allow for natural, varied prosody across different samples. Furthermore, for continuous traits like speaking rate, the model learns the concept of speed, allowing it to generate speech that is faster or slower than anything it actually heard during training.
High Quality Maintained
Often, adding constraints to a model makes the output sound robotic. In subjective listening tests, users significantly preferred the controlled models for affect. When told to be "happy," listeners overwhelmingly identified it as such compared to a baseline. Crucially, the Mean Opinion Scores (MOS) remained high (~4.1 to 4.2), showing no degradation in naturalness compared to state-of-the-art unsupervised baselines.
Limitations and Future Work
While highly effective, the training process can be sensitive. If the "supervision weight" isn't tuned correctly, the model might ignore the labels entirely or the latent space might "collapse" into a single point. Additionally, the authors noted that the simple distributions used for the latent variables (like Gaussian or Laplace) might be too simple to capture the full complexity of human emotion.
Future directions include moving toward more expressive "priors" (the mathematical assumptions about the shape of the latent space) and exploring how to control even more subtle aspects of speech, such as sarcasm or intent.
Reference: Habib, R., et al. (2020). Semi-Supervised Generative Modeling for Controllable Speech Synthesis. Published as a conference paper at ICLR 2020.
Summary updated