TL;DR
This paper introduces a generic, two-step methodology for converting transliterated text (like "Greeklish") into phonetic representations. This approach aims to enable text-to-speech (TTS) systems to pronounce non-native script versions of words accurately, bridging the gap between informal online communication and standard pronunciation.
What Problem Does This Paper Address?
The proliferation of informal online communication, such as emails and instant messaging, has led to the widespread use of transliterated text. This phenomenon, exemplified by "Greeklish" (Greek words written using the Latin alphabet), arises from the lack of native character support in early software and the desire for faster, less formal communication. However, this creates a significant challenge for systems like Text-to-Speech (TTS) that rely on standard orthography to generate pronunciation. Existing methods for converting Greeklish to Greek are often rule-based or rely on regular expressions, which struggle with the inherent ambiguity and non-standard nature of transliteration (e.g., multiple ways to represent the Greek letter "theta"). This paper addresses the need for a robust, data-driven methodology to accurately map these transliterated words to their phonetic equivalents.
Methods
The proposed methodology is a two-step process:
- Pronunciations Manager: This module generates a list of candidate pronunciations for a given transliterated word. It employs a statistical pronunciation model based on Tri-Symbol Classification and Regression Trees (Tri-Symbol CART or TSC). The TSC model learns the relationship between a letter within its context (preceding and following letters) and the corresponding phonemes of the target language (Greek, in this case). It outputs probabilities for each letter-to-phoneme mapping. The process involves:
- Splitting the input word into tri-symbols.
- Using the TSC model to calculate probabilities for each sequence.
- Pruning low-probability results.
- Generating a ranked list of pronunciations.
- Best-Result Engine: This module selects the most appropriate pronunciation from the list generated by the Pronunciations Manager. It utilizes a set of dictionaries:
- A Greeklish dictionary.
- A Greek dictionary (stressed and unstressed versions).
- An English dictionary. The engine prioritizes direct matches in the Greeklish dictionary, then uses the ranked pronunciations from the manager, checking against the Greek dictionaries. If no match is found, it defaults to the English dictionary or the top pronunciation with stress normalization.
A key technical component is the Letters-to-phonemes Alignment process, which automatically creates a trellis net to find the optimal path mapping input letters to output phonemes using dynamic programming.
Datasets
- Training Data: A corpus of approximately 8504 unique Greeklish-to-Greek pronunciation pairs was manually created. This was split into a training set of 6000 pairs and a test set of 2504 pairs.
- Evaluation Data: The test set of 2504 unique Greeklish words and their corresponding pronunciations was used for evaluation.
- Dictionaries:
- Greeklish dictionary (8504 words).
- Greek dictionary (93K stressed words).
- Greek dictionary (93K unstressed words).
- English dictionary (120K words).
Key Insights and Ideas
- Phonetic Representation as Intermediate Layer: The core idea is to use phonetic strings as an intermediate representation, abstracting away from the orthographic differences between the source transliteration and the target native script. This is crucial for handling the inherent ambiguity of transliteration.
- Statistical Modeling for Ambiguity: Relying on statistical models like CART, rather than hardcoded rules, allows the system to learn and handle the probabilistic nature of transliteration, where a single transliterated sequence can have multiple plausible phonetic interpretations.
- Contextual Letter Analysis: The Tri-Symbol CART approach, considering the preceding and following letters, is vital for disambiguating letter-to-phoneme mappings, which is a significant challenge in transliteration.
- Multi-Dictionary Approach for Robustness: The Best-Result Engine's use of multiple dictionaries (Greeklish, Greek, English) makes the system more robust, capable of handling pure English words and providing fallback options when direct Greeklish mappings are not found.
Key Results
- The paper evaluated five different CART models. The "minimum error" model achieved the highest accuracy of 81.87% correct pronunciations.
- The "standard cart" model, while slightly less accurate at 81.07%, was chosen for its comparable performance with significantly fewer leaves (8338 vs. 9478), indicating better generalization.
- The Best-Result Engine, when configured to generate up to 10 pronunciations, achieved 84.5% valid pronunciations from the Greek dictionaries.
- With a single pronunciation configuration, the engine achieved 78.5% valid pronunciations.
- The average placement of correct pronunciations in the ranked list was consistently low (around 1.42/10), suggesting the models effectively rank probable pronunciations high.
Limitations and Future Work
- Word-Based System: The current system is word-based, meaning it does not exploit the broader context of sentences or discourse. This limits its ability to resolve ambiguities that are only clear in context.
- Language Detection: The system assumes the input is Greeklish. A normalization step for language detection is suggested to prevent English text from being incorrectly processed as Greeklish.
- Text vs. Word-Based: Future work could involve developing text-based models to better leverage contextual information.
- Statistical Models for Output Processing: Further improvement could come from using statistical models to process the output of the pronunciations manager more elegantly.
Summary updated