EfficientTTS 2: Variational End-to-End Text-to-Speech Synthesis and Voice Conversion

Chenfeng Miao, Qingying Zhu, Minchuan Chen, Jun Ma, Shaojun Wang, Jing Xiao

Abstract

Text-to-speech (TTS) field is recently dominated by one-stage text-to-waveform models, in which the speech quality is significantly improved compared to two-stage models. However, the best-performing open-sourced one-stage model, the VITS, is not fully differentiable and suffers from relatively high computation costs. To address these issues, we propose EfficientTTS 2 (EFTS2), a fully differentiable end-to-end TTS framework that is highly efficient. Our method adopts an adversarial training process, with a differentiable aligner and a hierarchical-VAE-based waveform generator. The differentiable aligner is built upon the EfficientTTS. A hybrid attention mechanism and a variational alignment predictor are incorporated into our network to improve the expressiveness of the aligner. The use of the hierarchical-VAE-based waveform generator not only alleviates the one-to-many mapping problem in waveform generation but also allows the model to learn hierarchical and explainable latent variables that control different aspects of the generated speech. We also extend EFTS2 to the voice conversion (VC) task and propose EFTS2-VC, an end-to-end VC model that allows efficient and high-quality conversion. Experimental results suggest that the two proposed models match their strong counterparts in speech quality with a faster inference speed and smaller model size.
BibTeX
@article{miao2024efficienttts,
  author = {Chenfeng Miao and Qingying Zhu and Minchuan Chen and Jun Ma and Shaojun Wang and Jing Xiao},
  title = {{EfficientTTS 2: Variational End-to-End Text-to-Speech Synthesis and Voice Conversion}},
  journal = {IEEE ACM Trans. Audio Speech Lang. Process.},
  year = {2024},
  doi = {10.1109/taslp.2024.3369528},
}

Loading related content...

Figure