MuLanTTS: The Microsoft Speech Synthesis System for Blizzard Challenge 2023

Zhihang Xu, Shaofei Zhang, Xi Wang, Jiajun Zhang, Wenning Wei, Lei He, Sheng Zhao

TL;DR

MuLanTTS is Microsoft’s entry for the Blizzard Challenge 2023, delivering human-level naturalness in French text-to-speech (TTS) for long-form audiobooks and few-shot speaker adaptation. By augmenting a Conformer-based acoustic model with BERT-driven contextual, emotional, and linguistic representations, the system achieves mean opinion scores (MOS) of 4.3 and 4.5 across the challenge’s two tracks—performing on par with human recordings.

What Problem Does This Paper Address?

Standard neural TTS systems perform well on isolated, neutral sentences, but struggle when synthesizing long-form audiobooks. Key challenges include:

  1. Contextual prosody and expressiveness: Single-sentence models lack the context needed to produce coherent prosody across paragraphs and expressive dialogue versus neutral narration.
  2. French language nuances: Complex pronunciation behaviors, such as liaisons (linking final consonants to initial vowels) and polyphones (words spelled identically but pronounced differently), frequently trip up typical frontends.
  3. Low-resource adaptation: Adapting a voice model to a new speaker with only 2 hours of target audio while preserving high acoustic fidelity.

Methods

MuLanTTS builds on the DelightfulTTS framework and incorporates a series of modular enhancements across the pipeline:

Text ──> Multitask BERT Frontend (Liaison, Polyphone, POS) ──> Phonemes
               │
Text Context ──┼──> Contextual Encoder (Token + Sentence-level GRU) ──┐
               │                                                      │
Dialogue Context ─> Emotion Encoder (BERT-based CSE) ───────────────┼──> Conformer Acoustic Model ──> HiFi-GAN ──> Audio
                                                                     │    (Variance Adaptor: GST,
                                      Narration/Dialogue ID & Spk ID ─┘     Pitch, Duration, Prosody)
  • Linguistic Multitasking Frontend: Rather than relying on simple rule-based phonetizers, the team fine-tuned a French BERT model simultaneously on part-of-speech (POS) tagging (Universal Dependencies TreeBank), expert-labeled liaison prediction (binary classification), and polyphone disambiguation (labeled using OpenAI’s text-davinci-003).
  • Hierarchical Contextual Encoder: Captures both fine-grained token-level syntactic/semantic features and paragraph-level sentence embeddings, aggregating surrounding sentences via a GRU to provide historical and forward-looking prosodic context.
  • Context-Aware Emotion Encoder: Extracts a Context Sentence Embedding (CSE) from dialogue using a BERT model fine-tuned on web novels labeled with emotions by text-davinci-003. For narration, this representation is zero-initialized.
  • Conformer-Based Acoustic Model: A non-autoregressive acoustic backbone with 6 Conformer blocks. Its variance adaptor predicts phone duration, pitch, utterance-level Global Style Tokens (GST), and phone-level prosody embeddings.
  • Speech Enhancement & Vocoder: To prevent ambient noise in raw audiobook recordings from degrading synthesis quality, training audio was pre-processed using a deep speech enhancement model before training a 24 kHz HiFi-GAN vocoder (subsequently downsampled to 22.05 kHz).

Datasets

  • FH1 (Hub Task): ~50 hours of paired 22.05 kHz French audiobook recordings from a single female speaker.
  • FS1 (Spoke Task): 2 hours of paired 22.05 kHz speech from a second female French speaker, adapted from an internal 100+ hour multi-speaker French corpus (53 speakers; 25 female, 28 male).
  • Text & Auxiliary Corpora: French TreeBank (~14,000 sentences for POS tagging), ~1,600 sentences of expert-annotated liaison data, and web-scraped French novel excerpts labeled with GPT-3.5 (text-davinci-003) for polyphones and dialogue emotions.

Key Insights and Ideas

  • Continuous context beats discrete tags: Replacing discrete emotion labels with continuous implicit sentence embeddings (CSE) derived from BERT provided smoother, intensity-aware emotional control during synthesis.
  • Denoising is critical for training stability: Raw audiobooks often vary in recording channels and background noise. Running speech enhancement before training both the acoustic model and the vocoder prevented the synthesizer from learning or amplifying acoustic artifacts.
  • Multitask learning resolves phonetic ambiguity: French liaison prediction heavily correlates with grammatical structure; jointly predicting POS tags and liaison status within a single transformer model yielded robust pronunciations.

Key Results

  • Naturalness (MOS & MUSHRA):
  • FH1 (Hub): Achieved an MOS of 4.3, closely approaching the ground-truth recording score of 4.4 and ranking highest among all submitted synthetic systems. On MUSHRA, it scored 0.75 (recordings scored 0.94).
  • FS1 (Spoke): Achieved an MOS of 4.5 (matching ground-truth recordings at 4.5, with even lower variance: std 0.70 vs. 0.77). Its MUSHRA score was 0.76 (recordings scored 0.95), leading all submitted TTS systems.
  • Speaker Similarity (SMOS):
  • FH1: Scored 3.4, matching the ground-truth reference score.
  • FS1: Scored 4.3, statistically outperforming the original recording score (4.0).
  • Intelligibility: On Semantically Unpredictable Sentences (SUS), MuLanTTS achieved the second-lowest Word Error Rate (10.1%, just behind the top score of 9.5%). On homographs, it attained nearly 80% accuracy.

Limitations and Future Work

  • Untested Long-Form Paragraphs: Although the architecture is tailored for paragraph-level synthesis, the challenge’s formal listening tests primarily evaluated isolated sentence samples. Full paragraph-level evaluations remain future work.
  • Homograph Coverage: The authors noted that over 60% of test homograph types were not covered by their training set or dictionary, leaving clear room to scale up pronunciation modeling.
  • Complex Dialogue Nuance: More fine-grained modeling is needed for highly dynamic, expressive multi-turn dialogue interactions.

Summary updated

Abstract

In this paper, we present MuLanTTS, the Microsoft end-to-end neural text-to-speech (TTS) system designed for the Blizzard Challenge 2023. About 50 hours of audiobook corpus for French TTS as hub task and another 2 hours of speaker adaptation as spoke task are released to build synthesized voices for different test purposes including sentences, paragraphs, homographs, lists, etc. Building upon DelightfulTTS, we adopt contextual and emotion encoders to adapt the audiobook data to enrich beyond sentences for long-form prosody and dialogue expressiveness. Regarding the recording quality, we also apply denoise algorithms and long audio processing for both corpora. For the hub task, only the 50-hour single speaker data is used for building the TTS system, while for the spoke task, a multi-speaker source model is used for target speaker fine-tuning. MuLanTTS achieves mean scores of quality assessment 4.3 and 4.5 in the respective tasks, statistically comparable with natural speech while keeping good similarity according to similarity assessment. The excellent quality and similarity in this year’s new and dense statistical evaluation show the effectiveness of our proposed system in both tasks.
BibTeX
@inproceedings{xu2023mulantts,
  author = {Zhihang Xu and Shaofei Zhang and Xi Wang and Jiajun Zhang and Wenning Wei and Lei He and Sheng Zhao},
  title = {{MuLanTTS: The Microsoft Speech Synthesis System for Blizzard Challenge 2023}},
  booktitle = {Blizzard Challenge},
  year = {2023},
  url = {https://openreview.net/forum?id=i3zBbm8Ckb},
}
Figure