TL;DR
MuLanTTS is Microsoft’s entry for the Blizzard Challenge 2023, delivering human-level naturalness in French text-to-speech (TTS) for long-form audiobooks and few-shot speaker adaptation. By augmenting a Conformer-based acoustic model with BERT-driven contextual, emotional, and linguistic representations, the system achieves mean opinion scores (MOS) of 4.3 and 4.5 across the challenge’s two tracks—performing on par with human recordings.
What Problem Does This Paper Address?
Standard neural TTS systems perform well on isolated, neutral sentences, but struggle when synthesizing long-form audiobooks. Key challenges include:
- Contextual prosody and expressiveness: Single-sentence models lack the context needed to produce coherent prosody across paragraphs and expressive dialogue versus neutral narration.
- French language nuances: Complex pronunciation behaviors, such as liaisons (linking final consonants to initial vowels) and polyphones (words spelled identically but pronounced differently), frequently trip up typical frontends.
- Low-resource adaptation: Adapting a voice model to a new speaker with only 2 hours of target audio while preserving high acoustic fidelity.
Methods
MuLanTTS builds on the DelightfulTTS framework and incorporates a series of modular enhancements across the pipeline:
Text ──> Multitask BERT Frontend (Liaison, Polyphone, POS) ──> Phonemes
│
Text Context ──┼──> Contextual Encoder (Token + Sentence-level GRU) ──┐
│ │
Dialogue Context ─> Emotion Encoder (BERT-based CSE) ───────────────┼──> Conformer Acoustic Model ──> HiFi-GAN ──> Audio
│ (Variance Adaptor: GST,
Narration/Dialogue ID & Spk ID ─┘ Pitch, Duration, Prosody)
- Linguistic Multitasking Frontend: Rather than relying on simple rule-based phonetizers, the team fine-tuned a French BERT model simultaneously on part-of-speech (POS) tagging (Universal Dependencies TreeBank), expert-labeled liaison prediction (binary classification), and polyphone disambiguation (labeled using OpenAI’s
text-davinci-003). - Hierarchical Contextual Encoder: Captures both fine-grained token-level syntactic/semantic features and paragraph-level sentence embeddings, aggregating surrounding sentences via a GRU to provide historical and forward-looking prosodic context.
- Context-Aware Emotion Encoder: Extracts a Context Sentence Embedding (CSE) from dialogue using a BERT model fine-tuned on web novels labeled with emotions by
text-davinci-003. For narration, this representation is zero-initialized. - Conformer-Based Acoustic Model: A non-autoregressive acoustic backbone with 6 Conformer blocks. Its variance adaptor predicts phone duration, pitch, utterance-level Global Style Tokens (GST), and phone-level prosody embeddings.
- Speech Enhancement & Vocoder: To prevent ambient noise in raw audiobook recordings from degrading synthesis quality, training audio was pre-processed using a deep speech enhancement model before training a 24 kHz HiFi-GAN vocoder (subsequently downsampled to 22.05 kHz).
Datasets
- FH1 (Hub Task): ~50 hours of paired 22.05 kHz French audiobook recordings from a single female speaker.
- FS1 (Spoke Task): 2 hours of paired 22.05 kHz speech from a second female French speaker, adapted from an internal 100+ hour multi-speaker French corpus (53 speakers; 25 female, 28 male).
- Text & Auxiliary Corpora: French TreeBank (~14,000 sentences for POS tagging), ~1,600 sentences of expert-annotated liaison data, and web-scraped French novel excerpts labeled with GPT-3.5 (
text-davinci-003) for polyphones and dialogue emotions.
Key Insights and Ideas
- Continuous context beats discrete tags: Replacing discrete emotion labels with continuous implicit sentence embeddings (CSE) derived from BERT provided smoother, intensity-aware emotional control during synthesis.
- Denoising is critical for training stability: Raw audiobooks often vary in recording channels and background noise. Running speech enhancement before training both the acoustic model and the vocoder prevented the synthesizer from learning or amplifying acoustic artifacts.
- Multitask learning resolves phonetic ambiguity: French liaison prediction heavily correlates with grammatical structure; jointly predicting POS tags and liaison status within a single transformer model yielded robust pronunciations.
Key Results
- Naturalness (MOS & MUSHRA):
- FH1 (Hub): Achieved an MOS of 4.3, closely approaching the ground-truth recording score of 4.4 and ranking highest among all submitted synthetic systems. On MUSHRA, it scored 0.75 (recordings scored 0.94).
- FS1 (Spoke): Achieved an MOS of 4.5 (matching ground-truth recordings at 4.5, with even lower variance: std 0.70 vs. 0.77). Its MUSHRA score was 0.76 (recordings scored 0.95), leading all submitted TTS systems.
- Speaker Similarity (SMOS):
- FH1: Scored 3.4, matching the ground-truth reference score.
- FS1: Scored 4.3, statistically outperforming the original recording score (4.0).
- Intelligibility: On Semantically Unpredictable Sentences (SUS), MuLanTTS achieved the second-lowest Word Error Rate (10.1%, just behind the top score of 9.5%). On homographs, it attained nearly 80% accuracy.
Limitations and Future Work
- Untested Long-Form Paragraphs: Although the architecture is tailored for paragraph-level synthesis, the challenge’s formal listening tests primarily evaluated isolated sentence samples. Full paragraph-level evaluations remain future work.
- Homograph Coverage: The authors noted that over 60% of test homograph types were not covered by their training set or dictionary, leaving clear room to scale up pronunciation modeling.
- Complex Dialogue Nuance: More fine-grained modeling is needed for highly dynamic, expressive multi-turn dialogue interactions.
Summary updated