TL;DR
This paper introduces Deep Voice, a production-quality text-to-speech (TTS) system built entirely from deep neural networks, enabling end-to-end speech synthesis. The system achieves real-time inference speeds, making it practical for real-world applications, and demonstrates impressive flexibility by being trainable from scratch on new datasets without extensive feature engineering.
What Problem Does This Paper Address?
Traditional text-to-speech (TTS) systems are complex, multi-stage pipelines that often rely on laborious feature engineering and domain expertise. Developing new TTS systems, especially for different voices or languages, is time-consuming and difficult. Furthermore, achieving real-time inference for high-quality synthesized speech has been a significant challenge, particularly with powerful but computationally intensive models like WaveNet. This paper aims to simplify TTS system development by creating an end-to-end neural system and to demonstrate that such systems can achieve real-time performance.
Methods
Deep Voice is composed of five main neural network components:
- Grapheme-to-Phoneme (G2P) Model: Converts written text into phonemes. The paper uses a GRU-based encoder-decoder architecture, similar to prior work but with specific architectural choices.
- Segmentation Model: Locates phoneme boundaries within audio. This model uses a Connectionist Temporal Classification (CTC) loss function with a convolutional recurrent neural network (CRNN) architecture, adapted from speech recognition. This component is used for training data annotation, not inference.
- Phoneme Duration Model: Predicts the duration of each phoneme.
- Fundamental Frequency (F0) Prediction Model: Predicts whether a phoneme is voiced and its fundamental frequency contour. The duration and F0 models are jointly trained using a single neural network.
- Audio Synthesis Model: Generates the final audio waveform. This component is a modified version of WaveNet, optimized for faster inference. The paper highlights using QRNNs for upsampling and conditioning, and implementing optimized inference kernels for both CPU and GPU.
A key aspect of Deep Voice is its minimal reliance on hand-engineered features. It primarily uses phonemes, stress annotations, phoneme durations, and normalized fundamental frequency, which can be derived from audio and transcripts with less effort than traditional TTS features.
Datasets
- Internal English Speech Database: Approximately 20 hours of speech data, segmented into 13,079 utterances, spoken by a professional female speaker.
- Blizzard 2013 Data: Used for evaluating audio synthesis results, specifically a subset of 20.5 hours segmented into 9,741 utterances.
- CMUDict: Used for training the grapheme-to-phoneme model.
Key Insights and Ideas
- End-to-End Neural TTS is Achievable: The paper demonstrates that all components of a TTS system can be replaced with neural networks, leading to a simpler and more flexible system.
- Real-time Inference with WaveNet: Deep Voice shows that WaveNet, despite its computational demands, can be optimized for faster-than-real-time inference, making high-quality neural TTS practical.
- Reduced Feature Engineering: By relying on learned features and minimal hand-crafted inputs (phonemes, durations, F0), Deep Voice significantly simplifies the process of training new TTS models for different voices and domains.
- Joint Training of Duration and F0: Combining the prediction of phoneme duration and fundamental frequency into a single model improves efficiency and potentially the coherence of these acoustic features.
- Trade-off Between Speed and Quality: The system allows for a tunable trade-off between synthesis speed and audio quality, enabling users to select the best configuration for their needs.
Key Results
- Segmentation: Achieved a phoneme pair error rate of 7% after 14,000 iterations.
- Grapheme-to-Phoneme: Achieved a phoneme error rate of 5.8% and a word error rate of 28.7% after 20,000 iterations.
- Phoneme Duration and F0: Achieved a mean absolute error of 38 milliseconds for duration and 29.4 Hz for fundamental frequency.
- Audio Synthesis: The best models produced high-quality, recognizable speech. The paper reports Mean Opinion Scores (MOS) for various configurations, with the highest-quality synthesized audio (using synthesized F0 and duration) achieving a MOS of 2.00±0.23, while models using ground truth F0 and duration achieved higher scores (up to 3.94±0.26).
- Inference Speed: Optimized CPU and GPU kernels achieved speeds up to 400x faster than previous implementations, enabling real-time or faster-than-real-time synthesis. For example, a 40-layer WaveNet model achieved 1X real-time speed on CPU.
Limitations and Future Work
- Limitations:
- The paper notes that while the system is trainable from scratch, the MOS scores for fully synthesized audio (including synthesized duration and F0) are still lower than ground truth, indicating that duration and F0 prediction are the main barriers to achieving human-like naturalness.
- The GPU implementation of persistent kernels is labor-intensive and specialized for specific model configurations.
- Future Work:
- Further improve inference performance through model quantization (GPU and CPU), and exploring architectures like Xeon Phi.
- Investigate removing the separation between stages by merging segmentation, duration, and F0 prediction directly into the audio synthesis model, creating a single end-to-end sequence-to-sequence model.
- Explore ways to improve voice naturalness, potentially through larger training datasets or generative modeling techniques for duration and F0.
Summary updated