TL;DR
Brahmi-Net is a comprehensive system for transliteration and script conversion across 18 languages and 306 language pairs of the Indian subcontinent. It tackles the challenge of linguistic diversity by leveraging unsupervised mining of parallel transliteration corpora and a hybrid approach for languages lacking direct data, making cross-lingual communication more accessible.
What Problem Does This Paper Address?
The Indian subcontinent is a linguistic melting pot, boasting a vast number of spoken languages that utilize a diverse array of scripts, most of which are derived from the ancient Brahmi script. This rich diversity, while fascinating, presents a significant hurdle for effective communication, information retrieval, and computational linguistics applications. Specifically, the paper addresses the critical need for robust systems that can accurately convert text between different scripts (script conversion) and represent sounds phonetically in a target script (transliteration). Existing systems often struggle with the sheer scale of language and script variations, the lack of readily available parallel corpora for training, and the nuances of phonetic representation across different languages.
Methods
Brahmi-Net employs a two-pronged approach:
- Script Conversion:
- Brahmi Family Scripts: This relies on the fact that Brahmi-derived scripts share similar character inventories and Unicode codepoint ranges. The system primarily maps Unicode codepoints between these scripts. A special handling is implemented for the Tamil script, which uses characters for unvoiced plosives to represent voiced/aspirated ones. For Sinhala, which has less coordinated Unicode ranges, an explicit character mapping is used.
- Romanization (ITRANS): The system uses an extended ITRANS scheme for converting Indic scripts to Roman characters. For non-Devanagari scripts, it uses Devanagari as a pivot script, leveraging the existing Devanagari-to-ITRANS converter.
- Transliteration:
- Unsupervised Mining of Parallel Corpora: For the majority of language pairs, parallel transliteration corpora are not publicly available. Brahmi-Net addresses this by mining such corpora from existing parallel translation corpora (like the ILCI corpus) using an unsupervised approach. This method models corpus generation as a process involving transliteration and non-transliteration, learning parameters via the EM procedure to extract transliteration pairs.
- Hybrid Approach: For language pairs where direct parallel corpora cannot be mined, a hybrid approach is used. If a source language cannot be statistically transliterated, it's first converted to a phonetically similar "bridge language" using the rule-based script conversion system, and then statistically transliterated to the target language. The reverse is done for target languages that cannot be statistically transliterated.
- Statistical Transliteration: Phrase-based machine translation models (specifically, the Moses SMT system) are trained on the mined parallel transliteration corpora to learn character sequence mappings.
Datasets
- Training Data: The system is trained on mined parallel transliteration corpora. The paper reports mining a total of 1.69 million word pairs across 110 language pairs.
- Evaluation Data: For evaluating transliteration accuracy, an internally available, manually created corpus of 1000 transliteration pairs for each language pair was used. This corpus, while not reflecting the full diversity of the mined data, serves as a benchmark.
Key Insights and Ideas
- Leveraging Script Similarities: The core idea of mapping Unicode ranges for Brahmi-derived scripts is a clever and efficient way to handle script conversion for a large number of languages, exploiting inherent structural similarities.
- Unsupervised Mining as a Game Changer: The ability to automatically mine transliteration corpora from existing translation data is a significant contribution, overcoming the bottleneck of manual corpus creation and enabling scalability to many language pairs.
- Hybrid Approach for Robustness: The hybrid transliteration strategy, using rule-based conversion as a bridge, is crucial for ensuring coverage even for languages with limited direct parallel data.
- ITRANS as a Practical Romanization Standard: The choice of ITRANS, with its phonetic intuition and Roman keyboard compatibility, makes the system accessible for users familiar with Roman scripts.
Key Results
- Coverage: Brahmi-Net supports 18 languages (13 Indo-Aryan, 4 Dravidian, and English) and 306 language pairs for statistical transliteration.
- Mined Corpora: 1.69 million word pairs were mined for 110 language pairs. Some pairs, like hin-pan, mar-guj, and kok-mar, yielded a high number of mined pairs (over 30,000 on average).
- Transliteration Accuracy: Table 2 shows top-1 accuracy ranging from 27.9% (mal-tam) to 68.2% (tel-mar), and top-5 accuracy ranging from 30.9% (mal-tam) to 87.5% (tel-mar).
- SMT Improvement: The system demonstrated a significant improvement in machine translation quality when used to transliterate out-of-vocabulary (OOV) words in SMT outputs, as measured by BLEU scores (e.g., urd-eng saw an increase from 21.0% to 21.59% PB-SMT output).
Limitations and Future Work
- Unaddressed Characters: The script conversion system acknowledges that some characters may not have direct equivalents in other scripts (e.g., the nukta character in Dravidian scripts), an issue not yet addressed.
- Hybrid Model Refinement: The authors suggest scope for improvement in the hybrid transliteration models.
- Richer Transliteration Models: Exploring more sophisticated transliteration models beyond phrase-based SMT is a natural next step.
- Expanding Language and Script Support: A long-term goal is to incorporate other major South Asian languages, especially those with non-Brahmi scripts or significantly different phonetic structures.
Summary updated