AdaSpeech: Adaptive Text to Speech for Custom Voice

Mingjian Chen, Xu Tan, Bohan Li, Yanqing Liu, Tao Qin, Sheng Zhao, Tie-Yan Liu

TL;DR

AdaSpeech is an adaptive text-to-speech (TTS) system designed to create high-quality "custom voices" using as little as one minute of target speech. By combining multi-level acoustic modeling with a parameter-efficient "Conditional Layer Normalization" strategy, it achieves superior voice similarity and naturalness while requiring only ~5KB of storage per new speaker.

What Problem Does This Paper Address?

Creating a "custom voice"—a digital twin of a specific person's voice—is a core feature of modern speech platforms (like Microsoft Azure or Google Cloud). However, doing this effectively at scale presents two major hurdles:

  1. Acoustic Mismatch: The "source" data used to train the base model is usually high-quality and consistent. In contrast, "target" users might record their adaptation data in noisy rooms, with different microphones, or with unique accents and prosodies. Standard models struggle to generalize to these diverse conditions.
  2. The Memory Explosion: If a service has millions of users, storing a full fine-tuned model (often hundreds of megabytes) for every single person is economically and technically impossible. Conversely, just changing a "speaker embedding" (a small ID vector) often results in poor voice similarity.

AdaSpeech aims to find the "Goldilocks zone": high-quality adaptation that is robust to different recording environments but uses a negligible amount of storage per user.

Methods

The researchers built AdaSpeech on the foundation of FastSpeech 2, a popular non-autoregressive TTS model. They introduced two primary innovations to tackle the challenges of custom voice generation.

AdaSpeech Architecture

1. Acoustic Condition Modeling

To handle varying recording environments and speaking styles, AdaSpeech models acoustic information at three different granularities:

  • Speaker Level: A standard speaker embedding captures the general "timbre."
  • Utterance Level: An acoustic encoder extracts a global vector from a reference speech sample to capture the overall environment and emotion of a specific sentence.
  • Phoneme Level: A second encoder extracts a sequence of vectors (one for each phoneme) to capture local nuances like specific accents or stresses. During inference, a "Phoneme Level Predictor" guesses these values based on the text.

2. Conditional Layer Normalization (CLN)

This is the key to AdaSpeech’s efficiency. Instead of fine-tuning the entire mel-spectrogram decoder, the authors modified the Layer Normalization blocks.

Conditional Layer Normalization

They use the speaker embedding to dynamically generate the "scale" and "bias" vectors for every layer norm in the decoder.

  • Fine-tuning: During adaptation, only the parameters that generate these scales and biases (and the speaker embedding) are updated.
  • Inference: These parameters are pre-calculated into tiny vectors, meaning the system only needs to store about 5,000 parameters per speaker.

Datasets

  • Pre-training: The model was trained on LibriTTS (586 hours, ~2,456 speakers), a large-scale corpus of English speech derived from audiobooks.
  • Adaptation/Evaluation: To test "cross-domain" performance (where the target voice sounds different from the training data), they used:
  • VCTK: 108 speakers with various English accents.
  • LJSpeech: A high-quality single-speaker dataset.

Key Insights and Ideas

  • Explicit is Better than Implicit: By explicitly feeding the model information about the acoustic environment (utterance/phoneme level), the model stops trying to "memorize" the noise or style of the training data. Instead, it learns a cleaner mapping from text to speech, making it more adaptable.
  • The Power of Layer Norm: The authors discovered that layer normalization parameters, though few in number, have a massive impact on the "style" of the hidden activations in a neural network. Modifying just these parts is enough to shift the entire model's output to a new speaker's voice.
  • Decoupling Training and Inference: By using a predictor for phoneme-level features during inference, the model can benefit from high-resolution acoustic data during training without needing a "reference" audio clip from the user every time they want to generate speech.

Key Results

  • Quality: AdaSpeech significantly outperformed baselines in Mean Opinion Score (MOS) for naturalness and Similarity MOS (SMOS). On the VCTK dataset, it achieved an SMOS of 3.96, compared to 3.34 for a baseline that only tuned speaker embeddings.
  • Efficiency: While fine-tuning a whole decoder requires 14.1 million parameters, AdaSpeech achieves better quality with only 5,000 (5K) speaker-specific parameters in inference.
  • Data Requirements: The system remains robust even with very little data. While quality is best at 20 sentences (~1 minute), it remains competitive down to 10 sentences.

Limitations and Future Work

  • Data Floor: The authors noted that performance drops off quickly when the adaptation data is fewer than 10 sentences.
  • Noisy Environments: While AdaSpeech handles different acoustic conditions better than previous models, extremely noisy or distorted "custom voice" recordings remain a challenge.
  • Future Directions: The team plans to explore adaptation using untranscribed data (where the user provides audio but no text) and further compressing the model for edge device deployment.

Summary updated

Abstract

Custom voice, a specific text to speech (TTS) service in commercial speech platforms, aims to adapt a source TTS model to synthesize personal voice for a target speaker using few speech from her/him. Custom voice presents two unique challenges for TTS adaptation: 1) to support diverse customers, the adaptation model needs to handle diverse acoustic conditions which could be very different from source speech data, and 2) to support a large number of customers, the adaptation parameters need to be small enough for each target speaker to reduce memory usage while maintaining high voice quality. In this work, we propose AdaSpeech, an adaptive TTS system for high-quality and efficient customization of new voices. We design several techniques in AdaSpeech to address the two challenges in custom voice: 1) To handle different acoustic conditions, we model the acoustic information in both utterance and phoneme level. Specifically, we use one acoustic encoder to extract an utterance-level vector and another one to extract a sequence of phoneme-level vectors from the target speech during pre-training and fine-tuning; in inference, we extract the utterance-level vector from a reference speech and use an acoustic predictor to predict the phoneme-level vectors. 2) To better trade off the adaptation parameters and voice quality, we introduce conditional layer normalization in the mel-spectrogram decoder of AdaSpeech, and fine-tune this part in addition to speaker embedding for adaptation. We pre-train the source TTS model on LibriTTS datasets and fine-tune it on VCTK and LJSpeech datasets (with different acoustic conditions from LibriTTS) with few adaptation data, e.g., 20 sentences, about 1 minute speech. Experiment results show that AdaSpeech achieves much better adaptation quality than baseline methods, with only about 5K specific parameters for each speaker, which demonstrates its effectiveness for custom voice. The audio samples are available at https://speechresearch.github.io/adaspeech/.
BibTeX
@inproceedings{chen2021adaspeech,
  author = {Mingjian Chen and Xu Tan and Bohan Li and Yanqing Liu and Tao Qin and Sheng Zhao and Tie-Yan Liu},
  title = {{AdaSpeech: Adaptive Text to Speech for Custom Voice}},
  booktitle = {International Conference on Learning Representations},
  year = {2021},
  url = {https://openreview.net/forum?id=SdWVXYdw2xq},
}
Figure