Abstract
This paper presents the text-to-speech (TTS) system submitted by Idiap Research Institute to the Blizzard Challenge 2023. Our system follows the conventional pipeline of text analysis, acoustic modeling (AM) and vocoding. For text analysis, open-source pretrained part-of-speech (POS) taggers and lemmatizers are utilized to provide more accurate grapheme-to-phoneme (G2P) conversion on top of the eSpeak backend. The rest of the system incorporates a fully diffusion-based approach which comprises a diffusion transformer-based acoustic model and FastDiff as the vocoder, both of which are trained only on the provided data to ensure high-quality synthesis. Our entry provides a baseline for the cascading diffusion AM-vocoder architecture since no extra design is adopted to enhance the naturalness of speech. Evaluation results have demonstrated high synthesis quality of our system and the effectiveness of the proposed phonemization pipeline.
Introduction
The Blizzard Challenge 2023 focused on high-quality French speech synthesis using audiobook data. French presents specific linguistic challenges, notably liaisons (the pronunciation of linking consonants between words) and heterophonic homographs (words with identical spelling but different pronunciations based on grammatical context, such as fils meaning "son" or "threads").
Standard open-source G2P tools often fail to capture these nuances. Furthermore, modern TTS research is often hindered by the need for massive, closed-source models and high-end compute. The Idiap entry aimed to build a high-quality, fully diffusion-based baseline using only the provided challenge data and accessible hardware (a single consumer GPU).
System Architecture
The Idiap system utilizes a three-stage pipeline: text analysis, acoustic modeling, and vocoding. The core innovation lies in the integration of linguistic context into the front-end and the use of a cascading diffusion architecture.
Text Analysis Front-End
To address the homograph problem, the system enhances the eSpeak backend with a refinement layer. Pretrained Part-of-Speech (POS) taggers (CamemBERT-FLAIR) and lemmatizers (spaCy) are used to identify the grammatical role of words. This context allows the system to look up correct pronunciations in a curated table of 813 French homographs, significantly improving phonemic accuracy where standard tools are "blind" to grammar.
Diffusion-Based Acoustic Model and Vocoder
The acoustic model (AM) is a Diffusion Transformer (DiT). It processes phoneme sequences through a transformer encoder and a variance adapter (predicting pitch, energy, and duration) to generate mel-spectrograms via a diffusion-based decoder. For the final waveform generation, the system employs FastDiff, a conditional diffusion-based vocoder known for its balance of synthesis quality and inference efficiency.
Experimental Setup
The models were trained exclusively on the Blizzard Challenge 2023 datasets:
- Hub Task (NEB Corpus): 51 hours of French audiobook recordings from a single female speaker.
- Spoke Task (AD Corpus): 2 hours of recordings from a second female speaker, used for speaker adaptation via simple fine-tuning.
Results and Discussion
Evaluation results highlight the effectiveness of the diffusion-based baseline and the custom text-processing pipeline:
- Synthesis Quality: The system achieved a Mean Opinion Score (MOS) of 3.8, ranking 7th out of 18 participants in the Hub task.
- Linguistic Accuracy: The custom text pipeline reached 83% accuracy on heterophonic homographs, a 17% improvement over the baseline reference system (BF) which relied solely on eSpeak.
- Efficiency: The system demonstrated a real-time factor (RTF) of 0.48 on an NVIDIA RTX 3090, enabling speech generation at twice the speed of real-time.
Despite high quality scores, the system showed a high Word Error Rate (WER) of 19.4%. This is attributed to the rapid speaking rate of the source audiobooks and potential phoneme-audio misalignments from the Montreal Forced Aligner (MFA).
Conclusion
The Idiap submission successfully demonstrates that a cascading diffusion architecture can serve as a robust baseline for high-quality French TTS. While the system excels at handling complex linguistic features like homographs, future work will focus on improving prosody modeling for increased naturalness and refining alignment techniques to reduce word error rates.
Summary updated