The Xiaomi-ASLP Text-to-speech System for Blizzard Challenge 2023

Yuepeng Jiang, Kun Song, Fengyu Yang, Lei Xie, Meng Meng, Yu Ji, Yujun WangBlizzard Challenge 2023 · DOI

TL;DR

This paper presents Xiaomi-ASLP's entry for the Blizzard Challenge 2023, focusing on a VITS-based end-to-end TTS system for French. They introduce a novel "style adaptor" to better model prosody and global style from audiobook data, and leverage audio super-resolution to achieve higher output quality.

What Problem Does This Paper Address?

The Blizzard Challenge aims to advance text-to-speech (TTS) technology by fostering research on high-quality TTS systems. This year's hub task specifically focused on building a single-speaker French TTS system using a provided audiobook corpus. Key challenges in TTS, especially with audiobook data, include achieving high speaker similarity, naturalness, and intelligibility. Audiobook data, while rich in expressiveness, can be difficult to model due to inherent variations in style and prosody without explicit annotations. Furthermore, achieving high-fidelity audio output, particularly at higher sampling rates, remains an ongoing challenge.

Methods

The core of the Xiaomi-ASLP system is built upon the VITS (Variational Inference with Adversarial Learning for End-to-End Text-to-Speech) framework. To enhance VITS, they introduce several modifications:

  • Duration Modeling: Instead of VITS's original duration alignment, they replace it with a length regulator and a duration predictor, similar to FastSpeech, to improve duration prediction and incorporate external duration information.
  • Style Adaptor: This is a key innovation. It comprises two parts:
  • Fine-grained Prosody Module: This module uses a prosody encoder and a normalizing flow to implicitly model scene-specific prosody patterns and learn the relationship between text and prosody distribution.
  • LM-based Global Style Embedding: They leverage a large language model (GPT-3) to automatically annotate the audiobook text data with style tags (e.g., neutral, happy, sad). These tags are then used to extract style embeddings, which condition the text encoder to learn global style.
  • Audio Super-resolution: The VITS model generates audio at 16kHz. To achieve the target 48kHz output for better quality, they employ DSPGAN, a GAN-based vocoder, for audio super-resolution. This is a strategic choice to avoid potential training difficulties and errors associated with training directly on higher-sampling-rate data.

Datasets

  • Training/Evaluation: The primary dataset is the Blizzard Challenge 2023 hub task corpus, consisting of over 50 hours of French audio from five audiobooks narrated by a female speaker. The audio is at 22050 Hz.
  • Text Front-end: The open-source eSpeak is used as the text front-end to convert text into phoneme sequences.

Key Insights and Ideas

  • Leveraging LLMs for Style Annotation: Using GPT-3 for automatic style annotation of audiobook text is a clever way to overcome the lack of explicit style labels, significantly reducing the difficulty of style modeling.
  • Implicit Prosody Modeling: The fine-grained prosody module, by implicitly modeling prosody without explicit annotations, offers a more practical approach for expressive TTS.
  • Audio Super-resolution as a Quality Booster: Employing a separate super-resolution module (DSPGAN) for upsampling the synthesized audio is an effective strategy to achieve high-fidelity output without complicating the core TTS model training.
  • Duration Predictor's Importance: Replacing VITS's original duration mechanism with a length regulator and predictor, inspired by FastSpeech, highlights the critical role of accurate duration modeling for TTS.

Key Results

  • Naturalness (MOS): The system achieved an average MOS score of 3.4, described as "acceptable but not satisfactory."
  • Speaker Similarity (SMOS): The system scored significantly lower than expected (mean score 2.3). The authors attribute this to variations in the speaker's timbre across different audiobooks and the speaker clustering performed during data preparation.
  • Intelligibility (WER): The system had a relatively high word error rate (mean 23%) on semantically unpredictable sentences, which the authors link to issues in the French text front-end processing, particularly with pause handling.
  • Homograph Intelligibility: The system achieved 67% pronunciation accuracy for homographs.

Limitations and Future Work

  • French Text Front-end: The authors acknowledge significant shortcomings in their French text front-end processing, especially concerning pause handling, which directly impacted naturalness and intelligibility.
  • Speaker Similarity: The lower-than-expected speaker similarity scores are attributed to variations in the speaker's timbre across audiobooks and the data filtering process.
  • Future Work: The authors plan to address the shortcomings in their French front-end processing, explore its impact on homograph pronunciation, and further investigate the factors contributing to lower speaker similarity scores.

Summary updated

BibTeX
@inproceedings{jiang2023xiaomi,
  author = {Yuepeng Jiang and Kun Song and Fengyu Yang and Lei Xie and Meng Meng and Yu Ji and Yujun Wang},
  title = {{The Xiaomi-ASLP Text-to-speech System for Blizzard Challenge 2023}},
  booktitle = {Blizzard Challenge},
  year = {2023},
  doi = {10.21437/blizzard.2023-16},
}
Figure