TL;DR
This paper presents a real-time, high-fidelity neural text-to-speech (TTS) system for Japanese, a pitch-accent language. They achieve this by combining a novel sequence-to-sequence acoustic model that leverages full-context linguistic labels with two efficient neural vocoders: WaveGlow and a proposed Single Gaussian WaveRNN (SG-WaveRNN). The system demonstrates impressive synthesis quality and real-time performance, paving the way for more natural and responsive TTS applications.
What Problem Does This Paper Address?
Traditional text-to-speech (TTS) systems, especially for languages like Japanese with pitch accents, often rely on complex pipelines. These pipelines involve separate models for predicting speech duration and acoustic features, which can lead to a loss of naturalness and introduce artifacts. While neural TTS has shown promise in improving quality by directly generating waveforms, existing methods often struggle with real-time performance, particularly for pitch-accent languages. Autoregressive models like WaveNet, while high-quality, are inherently slow due to their sequential nature. Furthermore, adapting sequence-to-sequence (seq2seq) models, which offer end-to-end training, to pitch-accent languages has been challenging, with previous attempts showing inferior results compared to traditional pipeline models. This paper aims to bridge this gap by developing a unified, real-time neural TTS system that addresses both the acoustic modeling challenges for pitch-accent languages and the speed limitations of neural vocoders.
Methods
The core of the proposed system lies in two main components:
- Sequence-to-Sequence Acoustic Model (AM) with Full-Context Labels: The authors adapt the Tacotron 2 architecture for Japanese. Crucially, instead of using character or phoneme/accentual-type sequences as input, they employ "full-context labels" extracted from a text analyzer. These labels provide richer linguistic information, including past and future phonetic contexts, which are vital for accurately modeling pitch accents. This seq2seq AM directly predicts mel-spectrograms from these labels.
- Real-Time Neural Vocoders: To generate speech waveforms from the predicted mel-spectrograms efficiently, two vocoders are investigated:
- WaveGlow: A flow-based, non-autoregressive model known for its parallel generation capabilities and high quality.
- Single Gaussian WaveRNN (SG-WaveRNN): A proposed modification of the WaveRNN vocoder. Vanilla WaveRNN uses a dual-softmax for 16-bit audio prediction, requiring two sampling steps per audio sample. SG-WaveRNN simplifies this by modeling the conditional distribution of audio samples with a single Gaussian, reducing the synthesis time by half while maintaining high fidelity.
Additionally, the paper incorporates time-invariant noise shaping into the vocoders. This technique helps mitigate the impact of prediction errors, particularly beneficial for autoregressive models like WaveRNN.
Datasets
The experiments were conducted using a Japanese female speech corpus.
- Training: 25,626 utterances (approximately 22 hours) were used for training.
- Testing: 20 utterances were used for evaluation.
The paper does not introduce a new benchmark dataset but rather uses a standard corpus for evaluating their proposed system.
Key Insights and Ideas
- Full-Context Labels are Crucial for Pitch-Accent Languages: The success of the seq2seq AM for Japanese hinges on using full-context labels. This highlights that simply encoding phonemes and accent types isn't enough; richer contextual information is needed for accurate prosody modeling in languages with pitch variations.
- SG-WaveRNN Offers a Speed-Quality Trade-off: By simplifying the output distribution of WaveRNN, SG-WaveRNN achieves significantly faster synthesis times compared to its vanilla counterpart, making it a strong candidate for real-time applications without a drastic drop in quality.
- Noise Shaping Benefits AR Vocoders: The inclusion of noise shaping demonstrably improved the synthesis quality of autoregressive vocoders (SG-WaveRNN, AR SG-WaveNet) by making them more robust to prediction errors. This technique was less beneficial, and even slightly detrimental, for the non-autoregressive WaveGlow.
- Simple PyTorch Implementations are Sufficient for Real-Time: The paper demonstrates that even with straightforward PyTorch implementations, the proposed seq2seq AM and WaveGlow vocoder can achieve real-time factors (RTFs) that are highly competitive, suggesting that complex, highly optimized low-level code might not always be a prerequisite for real-time performance.
Key Results
- Subjective Evaluation (MOS): Neural TTS systems using the seq2seq AM with full-context labels and either WaveGlow or SG-WaveRNN vocoders achieved MOS scores over 4.0, indicating high synthesis quality comparable to natural speech.
- Real-Time Factor (RTF):
- Seq2seq AM + WaveGlow: RTF of 0.16 (using a GPU).
- Seq2seq AM + SG-WaveRNN: Achieved half the synthesis time of vanilla WaveRNN.
- Autoregressive models like AR SG-WaveNet had significantly higher RTFs (e.g., 205), highlighting the speed advantage of the proposed methods.
- SG-WaveRNN Speed: The proposed SG-WaveRNN can synthesize continuous-valued speech waveforms with half the synthesis time of vanilla WaveRNN.
Limitations and Future Work
- SG-WaveRNN GPU Optimization: While SG-WaveRNN is faster than vanilla WaveRNN, achieving real-time synthesis with a GPU still requires optimized GPU kernel implementations, which are a focus for future work.
- Mobile CPU Performance: For real-time synthesis on mobile CPUs, further investigation into sparse and subscale modifications for SG-WaveRNN is suggested.
- Vocoder Training Data: The current TTS system trains vocoders on ground-truth mel-spectrograms. Future work aims to improve synthesized speech quality by using mel-spectrograms predicted by the AM for vocoder training, similar to Tacotron 2.
- Comparison with Other Models: The authors plan to compare their proposed vocoders with other state-of-the-art models like parallel WaveNet, LPCNet, and NSF.
- Cross-Lingual and Cross-Model Comparisons: Future work will also involve comparing the seq2seq AM with traditional pipeline models and exploring other seq2seq approaches for different languages.
Summary updated