Abstract
MuLanTTS is a high-fidelity French text-to-speech (TTS) system developed for the Blizzard Challenge 2023. It addresses the challenges of long-form audiobook synthesis by incorporating hierarchical contextual encoders and emotion-aware BERT models into a Conformer-based architecture. The system achieves human-like prosody and expressiveness, nearly matching original recordings in naturalness and speaker similarity.
Introduction
Synthesizing audiobooks presents unique challenges compared to standard single-sentence TTS. Listeners expect long-form prosody—coherence across paragraphs—and expressive variation to distinguish between narration and dialogue. The French language introduces additional linguistic hurdles, such as complex liaison rules and polyphones (words with identical spellings but different pronunciations). Furthermore, audiobook datasets often contain background noise or varying acoustic environments that can degrade training quality. MuLanTTS was designed to overcome these obstacles through advanced architectural components and a robust linguistic frontend.
Methods
MuLanTTS builds upon the DelightfulTTS framework, introducing several sophisticated modules to handle the complexities of French audiobooks.
Conformer-based Acoustic Model
The system replaces standard Transformers with Conformer blocks, which capture both global and local dependencies more effectively through a combination of self-attention and convolution.
Hierarchical Contextual and Emotion Modeling
To ensure paragraph-level coherence, a BERT-based contextual encoder extracts token-level semantic features and sentence-level embeddings from surrounding text. For expressiveness, the system uses "Sentence Embedding" (CSE) derived from a fine-tuned BERT to provide a smooth representation of emotional intensity, rather than relying on discrete emotion tags.
Linguistic Frontend and Style Control
The linguistic frontend utilizes BERT for multitask learning to predict French-specific phenomena like POS tagging and polyphone disambiguation. Additionally, the system employs rule-based segmentation to identify narration versus dialogue, feeding a specific ID into the variance adaptor to help the model switch styles appropriately.
Key Results
The system was evaluated through subjective listening tests, including Mean Opinion Score (MOS) and MUSHRA tests, on two tracks: FH1 (50 hours of high-quality data) and FS1 (2 hours for speaker adaptation).
In the FH1 track, MuLanTTS (identified as System F) achieved a MOS of 4.3, very close to the original recording's 4.4. In the FS1 adaptation task, the system achieved a Similarity MOS (SMOS) of 4.3, surpassing the reference recording's score of 4.0 in that specific test. Objective metrics also showed high performance, with a Word Error Rate (WER) of 0.101 on Semantically Unpredictable Sentences (SUS).
Conclusion
MuLanTTS demonstrates that integrating contextual awareness and implicit emotion encoding significantly enhances the naturalness of long-form speech synthesis. While the system achieved leading results in the Blizzard Challenge 2023, future work will focus on improving homograph handling—currently at 80% accuracy—and further refining the synthesis of highly complex, expressive dialogue in extended contexts.
Summary updated