TL;DR
This paper tackles the challenge of training high-quality neural text-to-speech (TTS) systems using low-quality, noisy public recordings, which are abundant but difficult to use. They propose meta-learning and novel speaker adaptation techniques to achieve comparable voice quality and speaker similarity with significantly less data than traditional methods.
What Problem Does This Paper Address?
Generating natural-sounding speech with neural TTS systems typically requires large datasets of clean, studio-recorded audio. However, a wealth of publicly available recordings (e.g., from interviews, TV shows) exists, but these are often noisy, contain disfluencies, and lack professional quality. Training TTS models on such data is challenging due to these imperfections, leading to degraded speech quality and speaker similarity. This paper aims to bridge this gap by developing methods to adapt existing TTS models to create high-quality voices from these low-quality, limited public recordings.
Methods
The core of the paper lies in adapting a multi-speaker Tacotron TTS system. They explore three main adaptation strategies:
- Speaker Embeddings:
- Lookup Table: A standard approach where each speaker is assigned a unique ID and a corresponding embedding.
- Speaker Verification (SV) System: Leveraging a pre-trained speaker verification model (originally for "Personalized Hey Siri") to extract speaker embeddings. This is novel as it uses a system trained for a different task and potentially on different data.
- Meta-Learning (Pretraining-Finetuning): This is a key contribution. Instead of training a speaker-dependent model from scratch on the target low-quality data, they first pretrain a multi-speaker Tacotron model on a large, clean dataset (VCTK). This pretraining learns general text-to-speech mappings. Then, they "fine-tune" this pretrained model on a small amount of the target speaker's low-quality data. This fine-tuning process is applied in three ways, corresponding to the speaker embedding methods:
onehot-pf: Pretraining with lookup table embeddings, then fine-tuning.sv-pf: Pretraining with SV embeddings, then fine-tuning.noid-pf: Pretraining without any explicit speaker encoder, then fine-tuning. This tests if a speaker encoder is even necessary for adaptation.
Datasets
- Target Speaker Data: 19 hours of low-quality, noisy public recordings from spontaneous interviews. A subset of 1.5 hours with stable performance was selected for adaptation experiments. This data was manually transcribed and denoised.
- Multi-speaker Pretraining Data:
- VCTK Corpus: A standard multi-speaker English corpus (around 40 hours) used for pretraining the multi-speaker Tacotron models.
- Siri Corpus: A high-quality, professional English male voice corpus (13 hours) used for comparison to establish a baseline for high-quality TTS.
Key Insights and Ideas
- Low-quality data is a significant hurdle: Even with a large amount of data (19 hours), training a speaker-dependent model on noisy public recordings results in substantially lower naturalness and speaker similarity compared to training on clean data.
- Speaker verification embeddings are viable: Using embeddings from a pre-trained speaker verification system (like the one for PHS) can yield comparable results to traditional lookup table embeddings for speaker adaptation.
- Meta-learning is effective for adaptation: Pretraining a multi-speaker model on clean data and then fine-tuning on a small amount of low-quality target data significantly improves speech quality and speaker similarity, and importantly, increases the Signal-to-Noise Ratio (SNR).
- Speaker encoder might not be strictly necessary for adaptation: The
noid-pfsystem, which adapts a Tacotron model without an explicit speaker encoder after pretraining, still shows promising results, suggesting that the fine-tuning process itself can learn speaker-specific characteristics.
Key Results
- A speaker-dependent model trained on 19 hours of low-quality data achieved a naturalness MOS of 3.56, significantly lower than a model trained on 13 hours of high-quality Siri data (MOS 4.25).
- Adaptation systems (e.g.,
sv-1.5h,onehot-1.5h) trained on VCTK + 1.5 hours of target data showed improved SNR compared to the speaker-dependent model, indicating cleaner output. - Meta-learning systems (
onehot-pf-1.5h,sv-pf-1.5h,noid-pf-1.5h) achieved naturalness and similarity MOS scores comparable to or better than the speaker-dependent model trained on the full 19 hours of low-quality data, while also achieving higher SNRs. For instance,sv-pf-1.5hachieved a naturalness MOS of 3.60 and similarity MOS of 3.69.
Limitations and Future Work
- Noise robustness: While adaptation improves SNR, the paper acknowledges that the system's robustness to higher noise conditions still needs investigation.
- Zero-shot learning: The authors express interest in exploring the system's performance in zero-shot learning scenarios (i.e., adapting to a speaker with even less data, perhaps just a few seconds).
- Corpus denoising: Further research into methods for denoising the low-quality target corpus itself is suggested as a potential avenue for improvement.
- Early stopping: Designing effective early stopping mechanisms during the fine-tuning process is identified as a future research direction.
Summary updated