Deep Voice: Real-time Neural Text-to-Speech

Sercan Ömer Arik, Mike Chrzanowski, Adam Coates, Greg Diamos, Andrew Gibiansky, Yongguo Kang, Xian Li, John Miller, Jonathan Raiman, Shubho Sengupta, Mohammad Shoeybi

TL;DR

This paper introduces Deep Voice, a production-quality text-to-speech (TTS) system built entirely from deep neural networks, enabling end-to-end speech synthesis. The system achieves real-time inference speeds, making it practical for real-world applications, and demonstrates impressive flexibility by being trainable from scratch on new datasets without extensive feature engineering.

What Problem Does This Paper Address?

Traditional text-to-speech (TTS) systems are complex, multi-stage pipelines that often rely on laborious feature engineering and domain expertise. Developing new TTS systems, especially for different voices or languages, is time-consuming and difficult. Furthermore, achieving real-time inference for high-quality synthesized speech has been a significant challenge, particularly with powerful but computationally intensive models like WaveNet. This paper aims to simplify TTS system development by creating an end-to-end neural system and to demonstrate that such systems can achieve real-time performance.

Methods

Deep Voice is composed of five main neural network components:

  • Grapheme-to-Phoneme (G2P) Model: Converts written text into phonemes. The paper uses a GRU-based encoder-decoder architecture, similar to prior work but with specific architectural choices.
  • Segmentation Model: Locates phoneme boundaries within audio. This model uses a Connectionist Temporal Classification (CTC) loss function with a convolutional recurrent neural network (CRNN) architecture, adapted from speech recognition. This component is used for training data annotation, not inference.
  • Phoneme Duration Model: Predicts the duration of each phoneme.
  • Fundamental Frequency (F0) Prediction Model: Predicts whether a phoneme is voiced and its fundamental frequency contour. The duration and F0 models are jointly trained using a single neural network.
  • Audio Synthesis Model: Generates the final audio waveform. This component is a modified version of WaveNet, optimized for faster inference. The paper highlights using QRNNs for upsampling and conditioning, and implementing optimized inference kernels for both CPU and GPU.

A key aspect of Deep Voice is its minimal reliance on hand-engineered features. It primarily uses phonemes, stress annotations, phoneme durations, and normalized fundamental frequency, which can be derived from audio and transcripts with less effort than traditional TTS features.

Datasets

  • Internal English Speech Database: Approximately 20 hours of speech data, segmented into 13,079 utterances, spoken by a professional female speaker.
  • Blizzard 2013 Data: Used for evaluating audio synthesis results, specifically a subset of 20.5 hours segmented into 9,741 utterances.
  • CMUDict: Used for training the grapheme-to-phoneme model.

Key Insights and Ideas

  • End-to-End Neural TTS is Achievable: The paper demonstrates that all components of a TTS system can be replaced with neural networks, leading to a simpler and more flexible system.
  • Real-time Inference with WaveNet: Deep Voice shows that WaveNet, despite its computational demands, can be optimized for faster-than-real-time inference, making high-quality neural TTS practical.
  • Reduced Feature Engineering: By relying on learned features and minimal hand-crafted inputs (phonemes, durations, F0), Deep Voice significantly simplifies the process of training new TTS models for different voices and domains.
  • Joint Training of Duration and F0: Combining the prediction of phoneme duration and fundamental frequency into a single model improves efficiency and potentially the coherence of these acoustic features.
  • Trade-off Between Speed and Quality: The system allows for a tunable trade-off between synthesis speed and audio quality, enabling users to select the best configuration for their needs.

Key Results

  • Segmentation: Achieved a phoneme pair error rate of 7% after 14,000 iterations.
  • Grapheme-to-Phoneme: Achieved a phoneme error rate of 5.8% and a word error rate of 28.7% after 20,000 iterations.
  • Phoneme Duration and F0: Achieved a mean absolute error of 38 milliseconds for duration and 29.4 Hz for fundamental frequency.
  • Audio Synthesis: The best models produced high-quality, recognizable speech. The paper reports Mean Opinion Scores (MOS) for various configurations, with the highest-quality synthesized audio (using synthesized F0 and duration) achieving a MOS of 2.00±0.23, while models using ground truth F0 and duration achieved higher scores (up to 3.94±0.26).
  • Inference Speed: Optimized CPU and GPU kernels achieved speeds up to 400x faster than previous implementations, enabling real-time or faster-than-real-time synthesis. For example, a 40-layer WaveNet model achieved 1X real-time speed on CPU.

Limitations and Future Work

  • Limitations:
    • The paper notes that while the system is trainable from scratch, the MOS scores for fully synthesized audio (including synthesized duration and F0) are still lower than ground truth, indicating that duration and F0 prediction are the main barriers to achieving human-like naturalness.
  • The GPU implementation of persistent kernels is labor-intensive and specialized for specific model configurations.
  • Future Work:
    • Further improve inference performance through model quantization (GPU and CPU), and exploring architectures like Xeon Phi.
  • Investigate removing the separation between stages by merging segmentation, duration, and F0 prediction directly into the audio synthesis model, creating a single end-to-end sequence-to-sequence model.
  • Explore ways to improve voice naturalness, potentially through larger training datasets or generative modeling techniques for duration and F0.

Summary updated

Abstract

We present Deep Voice, a production-quality text-to-speech system constructed entirely from deep neural networks. Deep Voice lays the groundwork for truly end-to-end neural speech synthesis. The system comprises five major building blocks: a segmentation model for locating phoneme boundaries, a grapheme-to-phoneme conversion model, a phoneme duration prediction model, a fundamental frequency prediction model, and an audio synthesis model. For the segmentation model, we propose a novel way of performing phoneme boundary detection with deep neural networks using connectionist temporal classification (CTC) loss. For the audio synthesis model, we implement a variant of WaveNet that requires fewer parameters and trains faster than the original. By using a neural network for each component, our system is simpler and more flexible than traditional text-to-speech systems, where each component requires laborious feature engineering and extensive domain expertise. Finally, we show that inference with our system can be performed faster than real time and describe optimized WaveNet inference kernels on both CPU and GPU that achieve up to 400x speedups over existing implementations.
BibTeX
@inproceedings{arik2017deep,
  author = {Sercan Ömer Arik and Mike Chrzanowski and Adam Coates and Greg Diamos and Andrew Gibiansky and Yongguo Kang and Xian Li and John Miller and Jonathan Raiman and Shubho Sengupta and Mohammad Shoeybi},
  title = {{Deep Voice: Real-time Neural Text-to-Speech}},
  booktitle = {International Conference on Machine Learning},
  year = {2017},
}
Figure