Experiments with Cross-lingual Systems for Synthesis of Code-Mixed Text

Sunayana Sitaram, Sai Krishna Rallabandi, Shruti Rijhwani, Alan W. BlackSSW 2016 · DOI

TL;DR

This paper tackles the challenge of synthesizing code-mixed text (where multiple languages are mixed within a single sentence) for Text-to-Speech (TTS) systems. They extend previous work by improving language identification and exploring which language's TTS system works best as a "target" for code-mixed speech, finding a strong listener preference for Hindi as the target for Hindi-English text.

What Problem Does This Paper Address?

Modern Text-to-Speech (TTS) systems are typically trained on monolingual data, assuming input text is in a single language and uses standard spelling. However, the rise of social media has led to widespread "code-mixing," where speakers seamlessly blend multiple languages within sentences. This poses a significant challenge for TTS systems, which need to: 1) identify all languages present, 2) determine which word belongs to which language, and 3) apply appropriate pronunciation rules. Furthermore, code-mixed text often uses non-standard spellings and scripts (e.g., Romanized Hindi), complicating pronunciation. This paper aims to build TTS systems that can effectively synthesize such code-mixed text, specifically focusing on Hindi-English and German-English.

Methods

The authors build upon their previous framework for code-mixed TTS. Their approach involves:

  • Language Identification (LID): They improve upon a dictionary-based LID method by incorporating a Hidden Markov Model (HMM) for German-English and a modified Maximum Entropy classifier (based on Gella et al.) for Hindi-English. These methods aim to identify the language of each word in a code-mixed sentence.
  • Spelling Normalization: For Hindi-English, they employ a simple but effective technique of replacing non-standard spellings with high-frequency correct spellings from a Hindi recipe corpus, using the SoundEx algorithm to find matches. German-English did not require this step as it used native scripts.
  • Cross-lingual Synthesis: The core idea is to use a TTS system trained on a single language (the "target" language) to synthesize code-mixed text. This involves:
  • Identifying the language of each word.
  • Transliterating Hindi words to Devanagari script (if needed).
  • Mapping phonemes from the identified language's frontend to the target language's frontend.
  • Synthesizing the entire sentence using the target language's TTS system.
  • Target Language Selection: A key contribution is investigating which language should serve as the target. They compare using Hindi vs. English for Hindi-English and German vs. English for German-English, based on listener preference.

Datasets

  • Training Data: The paper utilized existing TTS databases:
  • Hindi: A 2.5-hour database from a single native Hindi speaker.
  • English: CMU ARCTIC [16] data (35 minutes) from the same Hindi speaker, and separate English data (30-35 minutes) from the German speakers.
  • German: AHW (32 minutes) and FEM (53 minutes) databases from two native German speakers.
  • Test Data: Code-mixed test sentences were collected from:
  • A Hindi recipe website (Hindi-English, Romanized script).
  • Twitter (German-English, native scripts).
  • 15 sentences from each corpus were used for listening tests.
  • Weakly-labeled Data: For HMM training, 100,000 tokens each of German and English were used.
  • Development Data: 500 Hindi-English tweets were used for tuning a decoding parameter.

Key Insights and Ideas

  • LID is Crucial: The paper strongly emphasizes that accurate word-level language identification is a prerequisite for successful code-mixed TTS. Their HMM and MaxEnt approaches significantly outperform simpler dictionary-based methods.
  • Target Language Matters: For code-mixed speech, choosing the "matrix language" (the language whose syntax governs the sentence) as the target TTS system is not always optimal. Listener preference tests revealed a strong preference for Hindi as the target for Hindi-English, even when English might be considered the matrix language in some instances.
  • Script Differences Complicate Things: The use of Romanized Hindi, lacking standardized spellings, presents a significant hurdle that requires dedicated normalization steps, unlike code-mixed text using native scripts.
  • Phoneme Mapping is Key: The ability to map phonetic representations between languages is fundamental to cross-lingual synthesis, especially when direct phoneme matches aren't available.

Key Results

  • LID Accuracy: The HMM/MaxEnt LID systems achieved significantly higher accuracy (e.g., 96% for German-English) compared to the dictionary-based approach (e.g., 65% for German-English). Normalized Hindi-English LID also showed improvement (89% vs. 69%).
  • Listener Preference (Target Language):
    • For Hindi-English, listeners strongly preferred the Hindi TTS system as the target (76% preference).
  • For German-English, listeners preferred the English TTS system as the target (79% for AHW, 82% for FEM).
  • Listener Preference (Cross-lingual vs. Monolingual):
    • The cross-lingual Hindi-English system was strongly preferred over a monolingual Hindi system (81% vs. 16%).
  • For German-English, the cross-lingual system was preferred over monolingual systems, though the preference was less pronounced than for Hindi-English (41% vs. 30% for AHW, 46% vs. 35% for FEM).

Limitations and Future Work

  • Spelling Normalization: The current spelling normalization for Hindi-English is simple and doesn't consider context or pronunciation. Future work could explore using word vectors to find more contextually appropriate spelling variants.
  • Bilingual Systems: The paper focuses on cross-lingual systems (using a single-language TTS). Future work includes building true bilingual TTS systems that combine phonetic spaces and improve cross-language pronunciation mapping.
  • Database Design: The current systems are built using existing monolingual databases. Future research could involve designing databases specifically to handle code-mixing and foreign words.
  • Phonetic Differences: The authors suggest that the less pronounced preference for cross-lingual German-English might be due to smaller phonetic differences between German and English compared to Hindi and English.

Summary updated

BibTeX
@inproceedings{sitaram2016experiments,
  author = {Sunayana Sitaram and Sai Krishna Rallabandi and Shruti Rijhwani and Alan W. Black},
  title = {{Experiments with Cross-lingual Systems for Synthesis of Code-Mixed Text}},
  booktitle = {SSW},
  year = {2016},
  doi = {10.21437/ssw.2016-13},
}
Figure