TL;DR
This paper presents a novel cross-lingual approach to developing an HMM-based speech synthesis system for Malay, an under-resourced language. By leveraging existing English speech synthesis resources and adapting them to Malay, the researchers successfully created a high-quality synthetic Malay voice with impressive intelligibility and naturalness, even with a limited dataset.
What Problem Does This Paper Address?
Developing high-quality speech synthesis systems for any language typically requires extensive resources: large amounts of recorded speech, detailed phonetic transcriptions, and precise segmental labels (marking the start and end times of each sound). Many languages, like Malay, are "under-resourced," meaning these crucial datasets are scarce or non-existent. This lack of resources makes building sophisticated systems like HMM-based speech synthesis (which relies on statistical modeling of speech sounds) a significant challenge. Traditional methods for generating segmental labels are often language-specific and require deep linguistic knowledge and specialized tools that may not be available for under-resourced languages. This paper tackles this problem by proposing a method to overcome the data scarcity for Malay speech synthesis.
Methods
The core of this research lies in its cross-lingual approach. Instead of building a Malay-specific system from scratch, the authors adapted existing English speech synthesis resources. Here's how:
- Leveraging Festival TTS: They used the established Festival English Text-to-Speech (TTS) system to generate time-aligned phonetic transcriptions for Malay. This was achieved by creating a custom Malay grapheme-to-phoneme (G2P) database and using English Classification and Regression Trees (CART) models.
- Phonetic and Syllabic Similarities: The approach exploits the phonetic and syllabic similarities between English and Malay. Malay has a relatively phonetic spelling system and a limited set of phonemes that overlap significantly with English.
- Adapting English Models: They adapted the contextual factors (linguistic information about phonemes and their surroundings) and duration models from existing English voices within Festival. This bypasses the need to develop these complex components from scratch for Malay.
- HMM Training: The generated time-aligned transcriptions, combined with a limited Malay speech database, were then used to train Hidden Markov Models (HMMs) using the HTS toolkit.
Datasets
- Training Data: A small corpus of 1000 phonetically balanced Malay sentences was compiled from various sources (newspapers, textbooks, general reading materials). This resulted in a speech database of 2763 words.
- G2P Database: A limited G2P database was built for Festival, containing 2,763 words from 1,000 phonetically balanced sentences.
- English Resources: Existing English speech synthesis resources within the Festival system, including G2P databases, CART models, and voice databases (specifically US English voices like
voice_nitech_us_bdl_arctic_htsandvoice_nitech_us_slt_arctic_hts), were adapted. - Evaluation Data: 50 synthetic Malay utterances of varying lengths were generated for intelligibility and naturalness testing.
Key Insights and Ideas
- Cross-lingual transfer is viable for under-resourced languages: The paper demonstrates that by intelligently adapting resources from a well-resourced language (English), high-quality speech synthesis can be achieved for languages with limited data.
- Phonetic similarities are key: The success hinges on the significant overlap in phonetic and syllabic structures between Malay and English, allowing for effective adaptation of models.
- Limited data can yield high quality: Even with a relatively small voice database (1000 sentences), the resulting synthetic Malay speech achieved impressive intelligibility and naturalness. This suggests that the quality of recordings and the accuracy of phonetic representations are more critical than sheer data volume for HMM-based systems.
- Duration accuracy impacts naturalness: The study found that better alignment between synthetic and recorded speech durations led to higher intelligibility and naturalness. This highlights the importance of accurate phone duration modeling.
Key Results
- High Intelligibility: The synthetic Malay voices achieved very high intelligibility scores. For the male voice, 99.33% of utterances were correctly transcribed by evaluators, and for the female voice, it was 99.11%.
- Good Naturalness: The male synthetic voice was judged slightly more natural than the recorded male voices (+0.82 on a 5-point scale), while the female voice was slightly less natural (-0.48).
- Accurate Durations: The average durational difference between synthetic and recorded utterances was low, around 3.96% for male voices and 5.36% for female voices, indicating good temporal alignment.
- Shorter utterances performed better: Intelligibility and naturalness were slightly higher for shorter utterances, likely due to lower durational differences.
Limitations and Future Work
- Limited Database Size: The authors acknowledge that the voice database used was small (1000 sentences). While the results are promising, a larger database could potentially further improve quality.
- Female Voice Naturalness: The female synthetic voice was rated slightly less natural than the male voice. This is attributed to potential issues with phone duration information in the segmentation labels for the female voice.
- Future Work: The paper suggests that further research could focus on improving the accuracy of phone duration information and exploring more advanced techniques for cross-lingual adaptation to further enhance the quality of synthetic speech for under-resourced languages. The authors also point to the availability of audio samples for interested readers to evaluate the synthetic voices themselves.
Summary updated