Abstract
Custom voice, a specific text to speech (TTS) service in commercial speech platforms, aims to adapt a source TTS model to synthesize personal voice for a target speaker using few speech data. Custom voice presents two unique challenges for TTS adaptation: 1) to support diverse customers, the adaptation model needs to handle diverse acoustic conditions that could be very different from source speech data, and 2) to support a large number of customers, the adaptation parameters need to be small enough for each target speaker to reduce memory usage while maintaining high voice quality. In this work, we propose AdaSpeech, an adaptive TTS system for high-quality and efficient customization of new voices. We design several techniques in AdaSpeech to address the two challenges in custom voice: 1) To handle different acoustic conditions, we use two acoustic encoders to extract an utterance-level vector and a sequence of phoneme-level vectors from the target speech during training; in inference, we extract the utterance-level vector from a reference speech and use an acoustic predictor to predict the phoneme-level vectors. 2) To better trade off the adaptation parameters and voice quality, we introduce conditional layer normalization in the mel-spectrogram decoder of AdaSpeech, and fine-tune this part in addition to speaker embedding for adaptation. We pre-train the source TTS model on LibriTTS datasets and fine-tune it on VCTK and LJSpeech datasets (with different acoustic conditions from LibriTTS) with few adaptation data, e.g., 20 sentences, about 1 minute speech. Experiment results show that AdaSpeech achieves much better adaptation quality than baseline methods, with only about 5K specific parameters for each speaker, which demonstrates its effectiveness for custom voice.
1. Introduction
Modern text-to-speech (TTS) systems have reached high levels of naturalness but typically require extensive high-quality recordings. "Custom voice" services face the challenge of adapting these models to new speakers using very limited data (e.g., one minute of audio). This task is complicated by two factors: acoustic mismatch between user-provided audio and professional training data, and the need for extreme parameter efficiency to support millions of users at scale.
Existing adaptation methods often struggle with this trade-off, either requiring massive per-speaker storage or failing to capture the nuances of the target voice. AdaSpeech addresses these issues through multi-level acoustic modeling and a novel normalization-based adaptation technique.
2. Methodology
2.1 Overall Architecture
AdaSpeech is based on the FastSpeech 2 backbone, a non-autoregressive architecture known for its speed and quality. The system incorporates specialized modules to handle acoustic variability and speaker-specific adaptation.
2.2 Acoustic Condition Modeling
To handle diverse recording environments and prosodic styles, AdaSpeech decomposes acoustic information into three granularities:
- Speaker Level: Represented by standard speaker embeddings.
- Utterance Level: A global vector is extracted from a reference sample to capture the overall recording environment and style.
- Phoneme Level: A sequence of vectors captures local variations, such as specific sound pronunciations. During inference, an acoustic predictor estimates these vectors from phoneme hidden states.
2.3 Conditional Layer Normalization (CLN)
For efficient adaptation, AdaSpeech introduces Conditional Layer Normalization in the mel-spectrogram decoder. Instead of updating all weights, the model uses speaker embeddings to dynamically generate scale and bias parameters for the LayerNorm blocks. During fine-tuning, only the small linear layers responsible for these parameters are updated, reducing the per-speaker storage requirement to approximately 5KB (roughly 5,000 parameters).
3. Experiments and Results
The model was pre-trained on the LibriTTS dataset (586 hours, 2,456 speakers) and evaluated on VCTK and LJSpeech. These target datasets present significant acoustic differences from the source data, testing the model's robustness.
Key findings include:
- High Quality: AdaSpeech significantly outperformed speaker-embedding-only baselines and matched the quality of full-decoder fine-tuning in Mean Opinion Score (MOS) tests.
- Efficiency: The system achieves high-quality cloning with only ~5,000 parameters per speaker, compared to 14.1 million for full fine-tuning.
- Few-Shot Robustness: The system performs effectively with as few as 20 sentences (approx. 1 minute of audio), though quality degrades if data falls below 10 sentences.
4. Conclusion
AdaSpeech provides a scalable and high-quality solution for custom voice applications. By disentangling acoustic conditions from linguistic content and utilizing surgical adaptation via Conditional Layer Normalization, it achieves state-of-the-art performance with minimal per-speaker overhead. Future work includes exploring adaptation with untranscribed data and improving robustness to extremely noisy environments.
Summary updated