TL;DR
AdaSpeech is an adaptive text-to-speech (TTS) system designed to create high-quality "custom voices" using as little as one minute of target speech. By combining multi-level acoustic modeling with a parameter-efficient "Conditional Layer Normalization" strategy, it achieves superior voice similarity and naturalness while requiring only ~5KB of storage per new speaker.
What Problem Does This Paper Address?
Creating a "custom voice"—a digital twin of a specific person's voice—is a core feature of modern speech platforms (like Microsoft Azure or Google Cloud). However, doing this effectively at scale presents two major hurdles:
- Acoustic Mismatch: The "source" data used to train the base model is usually high-quality and consistent. In contrast, "target" users might record their adaptation data in noisy rooms, with different microphones, or with unique accents and prosodies. Standard models struggle to generalize to these diverse conditions.
- The Memory Explosion: If a service has millions of users, storing a full fine-tuned model (often hundreds of megabytes) for every single person is economically and technically impossible. Conversely, just changing a "speaker embedding" (a small ID vector) often results in poor voice similarity.
AdaSpeech aims to find the "Goldilocks zone": high-quality adaptation that is robust to different recording environments but uses a negligible amount of storage per user.
Methods
The researchers built AdaSpeech on the foundation of FastSpeech 2, a popular non-autoregressive TTS model. They introduced two primary innovations to tackle the challenges of custom voice generation.
1. Acoustic Condition Modeling
To handle varying recording environments and speaking styles, AdaSpeech models acoustic information at three different granularities:
- Speaker Level: A standard speaker embedding captures the general "timbre."
- Utterance Level: An acoustic encoder extracts a global vector from a reference speech sample to capture the overall environment and emotion of a specific sentence.
- Phoneme Level: A second encoder extracts a sequence of vectors (one for each phoneme) to capture local nuances like specific accents or stresses. During inference, a "Phoneme Level Predictor" guesses these values based on the text.
2. Conditional Layer Normalization (CLN)
This is the key to AdaSpeech’s efficiency. Instead of fine-tuning the entire mel-spectrogram decoder, the authors modified the Layer Normalization blocks.
They use the speaker embedding to dynamically generate the "scale" and "bias" vectors for every layer norm in the decoder.
- Fine-tuning: During adaptation, only the parameters that generate these scales and biases (and the speaker embedding) are updated.
- Inference: These parameters are pre-calculated into tiny vectors, meaning the system only needs to store about 5,000 parameters per speaker.
Datasets
- Pre-training: The model was trained on LibriTTS (586 hours, ~2,456 speakers), a large-scale corpus of English speech derived from audiobooks.
- Adaptation/Evaluation: To test "cross-domain" performance (where the target voice sounds different from the training data), they used:
- VCTK: 108 speakers with various English accents.
- LJSpeech: A high-quality single-speaker dataset.
Key Insights and Ideas
- Explicit is Better than Implicit: By explicitly feeding the model information about the acoustic environment (utterance/phoneme level), the model stops trying to "memorize" the noise or style of the training data. Instead, it learns a cleaner mapping from text to speech, making it more adaptable.
- The Power of Layer Norm: The authors discovered that layer normalization parameters, though few in number, have a massive impact on the "style" of the hidden activations in a neural network. Modifying just these parts is enough to shift the entire model's output to a new speaker's voice.
- Decoupling Training and Inference: By using a predictor for phoneme-level features during inference, the model can benefit from high-resolution acoustic data during training without needing a "reference" audio clip from the user every time they want to generate speech.
Key Results
- Quality: AdaSpeech significantly outperformed baselines in Mean Opinion Score (MOS) for naturalness and Similarity MOS (SMOS). On the VCTK dataset, it achieved an SMOS of 3.96, compared to 3.34 for a baseline that only tuned speaker embeddings.
- Efficiency: While fine-tuning a whole decoder requires 14.1 million parameters, AdaSpeech achieves better quality with only 5,000 (5K) speaker-specific parameters in inference.
- Data Requirements: The system remains robust even with very little data. While quality is best at 20 sentences (~1 minute), it remains competitive down to 10 sentences.
Limitations and Future Work
- Data Floor: The authors noted that performance drops off quickly when the adaptation data is fewer than 10 sentences.
- Noisy Environments: While AdaSpeech handles different acoustic conditions better than previous models, extremely noisy or distorted "custom voice" recordings remain a challenge.
- Future Directions: The team plans to explore adaptation using untranscribed data (where the user provides audio but no text) and further compressing the model for edge device deployment.
Summary updated