TL;DR
This paper introduces Mixture-TTS, a novel non-autoregressive speech synthesis model that improves text-to-spectrogram alignment using a mixture alignment mechanism. By incorporating explicit pitch and energy predictors and a post-net structure, Mixture-TTS achieves high-quality audio synthesis with better alignment and reconstruction capabilities compared to existing methods.
What Problem Does This Paper Address?
Traditional speech synthesis models, especially those based on deep learning, have made significant strides in generating natural-sounding speech. However, a persistent challenge lies in accurately aligning the input text (phoneme sequences) with the corresponding acoustic features (mel-spectrograms). This alignment is crucial for generating high-quality audio. Existing methods often struggle with "phoneme blurring" due to the lack of clear boundaries in mel-spectrograms, leading to suboptimal alignment. Furthermore, while autoregressive models produce high-quality audio, they suffer from slow training and inference speeds. Non-autoregressive models offer speed but can sometimes compromise quality or alignment accuracy. This paper aims to bridge this gap by proposing a non-autoregressive model that enhances alignment and improves mel-spectrogram reconstruction for better audio quality.
Methods
The proposed model, Mixture-TTS, builds upon the FastSpeech2 architecture and incorporates several key innovations:
- Mixture Alignment Mechanism: Inspired by PortaSpeech, Mixture-TTS employs a linguistic encoder that utilizes a "mixture alignment" mechanism. This approach combines soft phoneme-level alignment with hard word-level alignment. This explicitly extracts word-level semantic information, addressing the boundary blurring issue in phoneme alignment.
- Pitch and Energy Predictors: To further optimize the rhythmic information of the synthesized audio, Mixture-TTS introduces dedicated predictors for pitch and energy. These predictors capture and utilize variance information from real audio during training, providing richer prosodic details during inference.
- Post-Net Structure: To enhance the reconstruction capability of the mel-spectrogram, a post-net module is integrated. This post-net is a five-layer 1D convolutional network that refines the mel-spectrogram output from the decoder, aiming to improve the accuracy and detail of the acoustic features.
- Two-Stage Training: The model follows a two-stage training scheme, common in modern TTS systems. The first stage involves training the acoustic model (Mixture-TTS) to convert text to mel-spectrograms, and the second stage uses a pre-trained HiFi-GAN vocoder to convert mel-spectrograms into audible waveforms.
Datasets
The experiments were conducted on two publicly available datasets:
- AISHELL3: A large-scale Chinese Mandarin speech dataset featuring 88,035 audio clips from 218 speakers, totaling approximately 85 hours.
- LJSpeech: A widely used English speech dataset comprising 13,100 audio clips from a single speaker, with a total duration of about 24 hours.
Both datasets were preprocessed, and text sequences were converted to phoneme sequences using pypinyin and g2p_en libraries.
Key Insights and Ideas
- Explicit Word-Level Alignment is Key: The paper strongly suggests that explicitly modeling word-level information alongside phoneme-level information is crucial for overcoming the limitations of purely phoneme-based alignment in speech synthesis.
- Prosody Matters: The inclusion of dedicated pitch and energy predictors significantly contributes to the rhythmic quality of the synthesized speech, highlighting the importance of capturing these prosodic features beyond just basic phoneme durations.
- Post-Nets for Refinement: The post-net architecture proves effective in refining the mel-spectrogram, leading to better reconstruction and, consequently, improved audio quality. This indicates that even with advanced acoustic models, a dedicated refinement stage can yield tangible benefits.
- Trade-offs in Non-Autoregressive Models: While non-autoregressive models offer speed advantages, the paper demonstrates that careful architectural design, like Mixture-TTS, can achieve competitive quality and alignment without sacrificing too much speed.
Key Results
- Improved Alignment: Mixture-TTS shows better alignment information between text sequences and mel-spectrograms, as evidenced by higher SSIM scores (0.514 on AISHELL3, 0.508 on LJSpeech) and lower MCD values (17.065 on AISHELL3, 6.635 on LJSpeech) compared to baseline models like FastSpeech2 and PortaSpeech.
- High-Quality Audio: Subjective evaluations using MOS scores show that Mixture-TTS produces audio quality comparable to or better than other state-of-the-art models, achieving a MOS of 3.92 on AISHELL3 and 3.96 on LJSpeech.
- Effective Components: Ablation studies confirm the effectiveness of the proposed components. Removing pitch and energy predictors led to a significant drop in CMOS scores, indicating their importance for rhythmic quality. Similarly, removing the post-net and residual network also degraded performance.
- Efficiency: The model achieves a relatively fast inference speed (RTF of 0.2152 on AISHELL3 and 0.1761 on LJSpeech), positioning it as a competitive non-autoregressive option.
Limitations and Future Work
- F0 RMSE: While strong in many metrics, Mixture-TTS is noted to be slightly weaker than some other models in predicting the F0 fundamental frequency (higher F0 RMSE).
- Future Research Directions: The authors suggest exploring other generative models as a basis for future research, potentially integrating Mixture-TTS's alignment mechanisms into more advanced architectures. They also express interest in fully end-to-end TTS models that directly generate waveforms from text, which they plan to pursue in future work. The paper also notes that PortaSpeech uses a flow-based post-net, which differs from the CNN-based post-net used in Mixture-TTS, suggesting potential avenues for further architectural exploration.
Summary updated