Prosody-Aware Speecht5 for Expressive Neural TTS

Yan Deng, Long Zhou, Yuanhao Yi, Shujie Liu, Lei HeICASSP 2023 · OpenReview

Abstract

SpeechT5, a multimodal learning framework which explores encoder-decoder pre-training by leveraging both unlabeled speech and text, has been proven to be effective on a wide variety of speech processing tasks. In this paper, we enhance SpeechT5 by adding a new sub-task on prosody modeling (prosody-aware SpeechT5) for neural text-to-speech (TTS), which can improve the model capability to learn richer contextual representations through multi-task learning. In the prosody-aware SpeechT5 training framework, most modules in neural TTS can be pre-trained with large-scale unlabeled speech and text corpus, including encoder, decoder, and variance adaptor. Experimental results show that the proposed prosody-aware SpeechT5 is effective at improving the expressiveness of neural TTS1: 1) the CMOS (comparison mean opinion score) gain is 0.154 for texts from news domain and 0.114 for texts from audiobook domain; 2) the prosody related issues in synthetic speech are reduced by 19.02% in subjective evaluation.
BibTeX
@inproceedings{deng2023prosody,
  author = {Yan Deng and Long Zhou and Yuanhao Yi and Shujie Liu and Lei He},
  title = {{Prosody-Aware Speecht5 for Expressive Neural TTS}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2023},
  url = {https://openreview.net/forum?id=U4PpF3EVhf},
}

Loading related content...

Figure