SUTD-NUS System for Blizzard Challenge 2021

Mingyang Zhang, Xuehao Zhou, Kun Zhou, Rui Liu, Perry Lam, Berrak Sisman, Haizhou Li

TL;DR

The SUTD-NUS team presents their end-to-end Spanish text-to-speech (TTS) system for the Blizzard Challenge 2021. Their approach leverages a modified Tacotron2 architecture for acoustic feature generation and a MelGAN vocoder for waveform synthesis, achieving competitive results on both standard Spanish and mixed-language Spanish-English synthesis tasks.

What Problem Does This Paper Address?

The paper tackles the challenge of building high-quality text-to-speech (TTS) systems, specifically for Spanish. The Blizzard Challenge 2021 presented two distinct tasks: synthesizing standard Spanish speech (SH1) and synthesizing Spanish speech that includes a small number of English words (SS1). The core problem is to develop a robust TTS system that can accurately convert text into natural-sounding speech, handling linguistic nuances and, in the SS1 task, seamlessly integrating foreign language elements. Traditional TTS systems often involve complex, multi-stage pipelines with manual feature engineering. This work aims to demonstrate the effectiveness of end-to-end deep learning approaches for these specific challenges.

Methods

The SUTD-NUS system is built upon an end-to-end TTS framework. The core of their system is a modified Tacotron2 architecture, a well-established sequence-to-sequence model for TTS.

  • Acoustic Feature Generation: The system uses an encoder-decoder architecture with an attention mechanism.
  • Encoder: Converts input characters into a fixed-dimension text embedding. For the Spanish tasks, they treat Spanish punctuation marks like "¿" and "¡" as tone indicators, embedding them to enhance expressiveness.
  • Decoder: Attends to the encoder's output to reconstruct acoustic features (mel-spectral features). This decoder also incorporates a pre-net and LSTM layers, followed by a post-net for improved reconstruction.
  • Waveform Synthesis: A MelGAN neural vocoder is employed to generate the final speech waveform from the predicted acoustic features. MelGAN is chosen for its fast inference and high-quality output, utilizing a generative adversarial network (GAN) with a non-autoregressive feed-forward convolutional architecture.

The system architecture is designed to be flexible, using the same core components for both the SH1 and SS1 tasks.

Datasets

The primary dataset used for training and evaluation is the approximately 5-hour European Spanish female speaker data provided by the Blizzard Challenge 2021 organizers. This dataset was used for both the SH1 (Spanish-only) and SS1 (Spanish with English words) tasks.

Key Insights and Ideas

  • Handling Mixed-Language Speech: The system demonstrates that a single end-to-end architecture can effectively handle both standard Spanish and Spanish text containing English words, by treating the input as character sequences and leveraging the shared alphabet.
  • Expressiveness through Punctuation: Incorporating Spanish-specific punctuation ("¿", "¡") as tone embeddings proved to be a simple yet effective way to improve the expressiveness of the synthesized speech.
  • MelGAN for Efficiency and Quality: The choice of MelGAN as the neural vocoder highlights its capability to generate high-fidelity speech waveforms efficiently, a crucial aspect for real-time applications.
  • End-to-End Advantage: The paper reinforces the trend towards end-to-end TTS systems, reducing the need for manual feature engineering and simplifying the overall pipeline.

Key Results

The paper reports evaluation results from the Blizzard Challenge organizers.

  • SH1 Task (Spanish):
    • Naturalness: Achieved a Mean Opinion Score (MOS) of 2.17 ± 1.08, compared to natural speech at 4.21 ± 0.93.
  • Similarity: Achieved an average MOS of 2.81 ± 1.28 for similarity to the original speaker.
  • Intelligibility: Obtained a word error rate (WER) of 6.1% on the INT-Sharvard test and 14% on the INT-SUS test.
  • SS1 Task (Spanish-English):
    • Naturalness: Achieved a MOS of 2.28 ± 1.08, compared to natural speech at 4.29 ± 0.88.
  • Similarity: Achieved an average MOS of 2.76 ± 1.25 for similarity to the original speaker.
  • Acceptability of English Words: Achieved a MOS of 2.20 ± 1.08.

While the system's scores for naturalness and similarity are lower than natural speech, they are presented as competitive within the context of the challenge, demonstrating the effectiveness of their approach.

Limitations and Future Work

  • Naturalness and Similarity Gaps: The reported MOS scores for naturalness and similarity are significantly lower than those for natural speech. This indicates room for improvement in generating more human-like and speaker-faithful audio.
  • Intelligibility in Mixed Language: While the system handles mixed-language input, the intelligibility scores, particularly for the SS1 task, suggest that further refinement might be needed to ensure perfect comprehension of both Spanish and embedded English words.
  • Future Work: The paper does not explicitly detail future work, but the results suggest potential avenues for improvement in acoustic modeling, vocoder fine-tuning, and potentially exploring more advanced techniques for handling code-switching in TTS.

Summary updated

Abstract

This paper describes our effort to build the SUTD-NUS system for Blizzard Challenge 2021. The challenge has two tasks: 1) Hub task 2021-SH1: to build a Spanish text-to-speech (TTS) system using about 5 hours data from a European Spanish female speaker, and 2) Spoke task 2021-SS1: to build a TTS system that is able to synthesize the Spanish text containing a small amount of English words, using the same training data as Hub task 2021-SH1. Our submitted system is an end-to-end TTS structure that can generate acoustic features from text input. MelGAN neural vocoder are utilized to generate speech waveforms from acoustic features for both SH1 and SS1 tasks. Evaluation results provided by the challenge organizers demonstrate the effectiveness of our submitted TTS system.
BibTeX
@inproceedings{zhang2021sutd,
  author = {Mingyang Zhang and Xuehao Zhou and Kun Zhou and Rui Liu and Perry Lam and Berrak Sisman and Haizhou Li},
  title = {{SUTD-NUS System for Blizzard Challenge 2021}},
  booktitle = {Blizzard Challenge},
  year = {2021},
  doi = {10.21437/blizzard.2021-12},
}

Loading related content...

Figure