TL;DR
This paper tackles the challenge of synthesizing code-mixed text (where multiple languages are mixed within a single sentence) for Text-to-Speech (TTS) systems. They extend previous work by improving language identification and exploring which language's TTS system works best as a "target" for code-mixed speech, finding a strong listener preference for Hindi as the target for Hindi-English text.
What Problem Does This Paper Address?
Modern Text-to-Speech (TTS) systems are typically trained on monolingual data, assuming input text is in a single language and uses standard spelling. However, the rise of social media has led to widespread "code-mixing," where speakers seamlessly blend multiple languages within sentences. This poses a significant challenge for TTS systems, which need to: 1) identify all languages present, 2) determine which word belongs to which language, and 3) apply appropriate pronunciation rules. Furthermore, code-mixed text often uses non-standard spellings and scripts (e.g., Romanized Hindi), complicating pronunciation. This paper aims to build TTS systems that can effectively synthesize such code-mixed text, specifically focusing on Hindi-English and German-English.
Methods
The authors build upon their previous framework for code-mixed TTS. Their approach involves:
- Language Identification (LID): They improve upon a dictionary-based LID method by incorporating a Hidden Markov Model (HMM) for German-English and a modified Maximum Entropy classifier (based on Gella et al.) for Hindi-English. These methods aim to identify the language of each word in a code-mixed sentence.
- Spelling Normalization: For Hindi-English, they employ a simple but effective technique of replacing non-standard spellings with high-frequency correct spellings from a Hindi recipe corpus, using the SoundEx algorithm to find matches. German-English did not require this step as it used native scripts.
- Cross-lingual Synthesis: The core idea is to use a TTS system trained on a single language (the "target" language) to synthesize code-mixed text. This involves:
- Identifying the language of each word.
- Transliterating Hindi words to Devanagari script (if needed).
- Mapping phonemes from the identified language's frontend to the target language's frontend.
- Synthesizing the entire sentence using the target language's TTS system.
- Target Language Selection: A key contribution is investigating which language should serve as the target. They compare using Hindi vs. English for Hindi-English and German vs. English for German-English, based on listener preference.
Datasets
- Training Data: The paper utilized existing TTS databases:
- Hindi: A 2.5-hour database from a single native Hindi speaker.
- English: CMU ARCTIC [16] data (35 minutes) from the same Hindi speaker, and separate English data (30-35 minutes) from the German speakers.
- German: AHW (32 minutes) and FEM (53 minutes) databases from two native German speakers.
- Test Data: Code-mixed test sentences were collected from:
- A Hindi recipe website (Hindi-English, Romanized script).
- Twitter (German-English, native scripts).
- 15 sentences from each corpus were used for listening tests.
- Weakly-labeled Data: For HMM training, 100,000 tokens each of German and English were used.
- Development Data: 500 Hindi-English tweets were used for tuning a decoding parameter.
Key Insights and Ideas
- LID is Crucial: The paper strongly emphasizes that accurate word-level language identification is a prerequisite for successful code-mixed TTS. Their HMM and MaxEnt approaches significantly outperform simpler dictionary-based methods.
- Target Language Matters: For code-mixed speech, choosing the "matrix language" (the language whose syntax governs the sentence) as the target TTS system is not always optimal. Listener preference tests revealed a strong preference for Hindi as the target for Hindi-English, even when English might be considered the matrix language in some instances.
- Script Differences Complicate Things: The use of Romanized Hindi, lacking standardized spellings, presents a significant hurdle that requires dedicated normalization steps, unlike code-mixed text using native scripts.
- Phoneme Mapping is Key: The ability to map phonetic representations between languages is fundamental to cross-lingual synthesis, especially when direct phoneme matches aren't available.
Key Results
- LID Accuracy: The HMM/MaxEnt LID systems achieved significantly higher accuracy (e.g., 96% for German-English) compared to the dictionary-based approach (e.g., 65% for German-English). Normalized Hindi-English LID also showed improvement (89% vs. 69%).
- Listener Preference (Target Language):
- For Hindi-English, listeners strongly preferred the Hindi TTS system as the target (76% preference).
- For German-English, listeners preferred the English TTS system as the target (79% for AHW, 82% for FEM).
- Listener Preference (Cross-lingual vs. Monolingual):
- The cross-lingual Hindi-English system was strongly preferred over a monolingual Hindi system (81% vs. 16%).
- For German-English, the cross-lingual system was preferred over monolingual systems, though the preference was less pronounced than for Hindi-English (41% vs. 30% for AHW, 46% vs. 35% for FEM).
Limitations and Future Work
- Spelling Normalization: The current spelling normalization for Hindi-English is simple and doesn't consider context or pronunciation. Future work could explore using word vectors to find more contextually appropriate spelling variants.
- Bilingual Systems: The paper focuses on cross-lingual systems (using a single-language TTS). Future work includes building true bilingual TTS systems that combine phonetic spaces and improve cross-language pronunciation mapping.
- Database Design: The current systems are built using existing monolingual databases. Future research could involve designing databases specifically to handle code-mixing and foreign words.
- Phonetic Differences: The authors suggest that the less pronounced preference for cross-lingual German-English might be due to smaller phonetic differences between German and English compared to Hindi and English.
Summary updated