TL;DR
This paper introduces CML-TTS, a new multilingual text-to-speech (TTS) dataset derived from audiobooks in seven low-resource languages. This dataset, along with a trained YourTTS model, aims to advance research in multilingual TTS by providing a valuable resource for developing models that can synthesize speech in diverse languages.
What Problem Does This Paper Address?
The field of Text-to-Speech (TTS) has seen significant advancements thanks to deep learning, enabling the integration of traditional TTS modules into single, high-performing models. However, a major bottleneck for training these deep learning-based TTS models is the requirement for vast amounts of high-quality, transcribed audio data. Consequently, most current TTS models are heavily focused on English, which has abundant open-source datasets like LJSpeech and LibriTTS. While datasets like Multilingual LibriSpeech (MLS) exist for speech recognition, they are not directly suitable for TTS training due to differences in audio quality, sampling rates, and the absence of precise punctuation in transcripts. This gap limits the development and accessibility of TTS systems for low-resource languages, hindering the creation of truly multilingual virtual assistants and speech technologies.
Methods
The core contribution is the creation of the CML-TTS dataset. The authors adapted the Multilingual LibriSpeech (MLS) dataset by processing audiobooks from the LibriVox project. The data processing pipeline involved four key steps:
- Audio Download and Conversion: Original MP3 audio files were downloaded via the LibriVox API and converted to WAV format at a 24kHz sampling rate.
- Text Punctuation Retrieval: A novel method was developed to automatically add punctuation to the existing MLS text transcripts. This involved searching for equivalent sentences in punctuated textbooks and using Levenshtein distance for similarity matching, with a sliding window approach to find the best alignment.
- Audio Segmentation: Audio segments longer than 15 seconds were split into smaller segments (ideally two sentences) using text punctuation as a guide, employing the Aeneas tool for audio-text alignment.
- Text Validation: A Wav2Vec 2.0 XLSR Large model was used to transcribe the audio segments. The similarity between the transcribed text and the ground truth text was calculated, and segments with a similarity score below 0.9 were discarded to ensure data quality.
Additionally, the paper presents experiments using the YourTTS model, a multilingual zero-shot multi-speaker TTS model, trained on the CML-TTS dataset.
Datasets
- CML-TTS: The newly introduced dataset, comprising audiobooks in seven languages: Dutch, French, German, Italian, Polish, Portuguese, and Spanish. It's derived from the MLS dataset and adapted for TTS training. It contains approximately 3,233.43 hours of audio from 613 speakers.
- Multilingual LibriSpeech (MLS): The source dataset for CML-TTS.
- LibriTTS: Used in conjunction with CML-TTS for training the YourTTS model to improve generalization across a larger number of speakers.
- YourTTS Model Training Data: A combination of CML-TTS (3,176.13 hours) and LibriTTS (245.07 hours in English).
- Other Datasets Mentioned for Context/Comparison: VCTK, LJSpeech, Common Voice, TedX, VoxCeleb 2, M-AILABS.
Key Insights and Ideas
- Bridging the TTS Data Gap: The creation of CML-TTS directly addresses the critical need for diverse, high-quality multilingual TTS datasets, which are scarce compared to English.
- Automated Punctuation for TTS: The developed text punctuation retrieval method is a novel contribution, crucial for making existing text data suitable for TTS synthesis.
- Quality Control is Paramount: The multi-step validation process, including STT-based transcription and similarity checks, highlights the importance of rigorous quality control for TTS datasets.
- YourTTS Effectiveness: The paper demonstrates that the YourTTS model, trained on CML-TTS, can achieve good performance in synthesizing speech for multiple languages, including those not explicitly seen during its initial pre-training.
- Dataset Balance Considerations: While the dataset has a good number of speakers, the authors note an imbalance in total hours between male and female speakers, which is a common challenge in speech datasets.
Key Results
- The CML-TTS dataset contains approximately 3,233.43 hours of audio across seven languages (Dutch, French, German, Italian, Polish, Portuguese, Spanish) from 613 speakers, with a 24kHz sampling rate.
- Experiments with the YourTTS model showed promising results, with Speaker Encoder Cosine Similarity (SECS) scores generally above 0.77 for both seen and unseen speakers across languages. Spanish achieved the highest SECS scores (0.8536 for seen, 0.8180 for unseen speakers).
- Word Error Rate (WER) and Character Error Rate (CER) metrics for synthesized speech validation were also reported, with Spanish showing the best performance and Portuguese the worst, potentially due to orthographic changes in the language.
Limitations and Future Work
- Gender Imbalance: The dataset has a significant imbalance in the total duration of audio between male (2,278 hours) and female (897 hours) speakers.
- Portuguese Orthography: The performance in Portuguese was affected by the use of older book texts that predate a 1990 orthographic agreement, leading to potential inconsistencies.
- No Language Model for STT: The STT model used for validation did not employ a language model, which might have influenced the WER/CER results.
- Future Work: The authors suggest that this dataset opens up new research possibilities for multilingual TTS models and code-switching capabilities. Further work could involve addressing the gender imbalance and exploring more advanced multilingual model architectures.
Summary updated