TL;DR
This paper tackles the challenging task of converting romanized text from languages with native scripts (like Bengali and Hindi) back to their original script, especially when those texts are "code-mixed" with English. They propose a novel, three-component system that achieves state-of-the-art results, outperforming existing tools like Google Translate.
What Problem Does This Paper Address?
Many languages, particularly in South Asia, have their own native scripts but are often written using the Roman alphabet, especially in informal digital communication like social media. This "ad-hoc romanization" is a result of keyboard limitations or user preference. The problem is compounded by "code-mixing," where words from another language (typically English) are interspersed within the text. This makes it difficult for standard Natural Language Processing (NLP) tools, which are trained on native script text, to process these messages. Deromanization, the process of converting this romanized text back to its native script, is crucial for enabling NLP tasks like sentiment analysis and opinion mining. Existing approaches often tackle romanization and code-mixing separately, leading to suboptimal performance.
Methods
The authors propose a joint approach that combines three key components:
- Language Identification: This module identifies which language each word in the code-mixed text belongs to. It uses an encoder-decoder model based on bi-directional LSTMs, which has shown superior performance for this task compared to other sequence labeling models.
- Back-Transliteration: This component converts romanized words into their native script. Instead of relying on a single system, they pool the top predictions from three diverse transliteration systems: Sequitur (a generative joint n-gram transducer), DTLM (a discriminative transducer), and OpenNMT (a neural machine translation tool). They also leverage target word lists and character language models to improve accuracy.
- Sequence Prediction: This module takes the candidates from the back-transliteration step and uses a Hidden Markov Model (HMM) to select the most likely sequence of native script words. It considers both the transliteration scores and word trigram language model scores to account for word context within the sentence, addressing the limitations of systems that process words in isolation.
Datasets
The paper utilizes several datasets:
- NEWS 2018 shared task: Used for training back-transliteration systems for Bengali and Hindi.
- FIRE 2014 shared task: Used for language identification and for creating back-transliteration test sets by extracting romanized Bengali and Hindi word tokens.
- Newly created Bengali development set: Collected from Facebook groups and pages, manually deromanized, and used for developing and tuning the system independently from test sets. This dataset has a lower percentage of English words compared to the FIRE 2014 datasets.
- Bengali Wikipedia, Bengali news corpus, and Hindi news corpus: Used to derive character language models and target word lists for DTLM.
- Manually-created transliterations of frequent Bengali words and automatically generated romanizations of 50,000 Bengali words from Wikipedia: Used to augment training data for Bengali back-transliteration.
Key Insights and Ideas
- Jointly addressing romanization and code-mixing is crucial: The paper demonstrates that treating these two problems together leads to significantly better results than tackling them in isolation.
- Pooling diverse transliteration systems improves robustness: By combining predictions from generative, discriminative, and neural models, the system can leverage different strengths and overcome individual weaknesses.
- Sequence prediction is vital for context: Simply deromanizing each word independently is insufficient. Incorporating context through language models helps disambiguate and select the correct transliteration.
- Synthetic data generation can boost performance: Automatically generating romanized data for training neural transliteration models significantly improved accuracy.
Key Results
- The proposed joint system achieves state-of-the-art results on deromanization of code-mixed texts for Bengali and Hindi.
- The system substantially outperforms Google Translate on both Bengali and Hindi test sets, with an average of 27% of Google Translate's errors attributed to deromanizing English words.
- For language identification, their encoder-decoder model achieved high accuracy (in the low 90s%), outperforming other systems.
- Back-transliteration results showed strong performance, with the neural system (OpenNMT) achieving the best top-1 accuracy on Bengali when augmented with synthetic data.
Limitations and Future Work
- Domain differences: The paper notes that the development set (social media) and test sets (search queries) have different domains, which can lead to discrepancies in performance. The social media domain exhibits more spelling variation.
- Future work: The authors plan to apply their approach to other languages and scripts that involve both code-mixing and romanization.
Summary updated