Machine Translation by Projecting Text into the Same Phonetic-Orthographic Space Using a Common Encoding

Amit Kumar, Shantipriya Parida, Ajay Pratap, A. K. SingharXiv 2023 · DOI · arXiv

TL;DR

This paper introduces a method to boost Neural Machine Translation (NMT) for low-resource Indian languages by projecting diverse scripts into a shared phonetic-orthographic space using WX notation. By converting different scripts (like Devanagari, Gurumukhi, and Perso-Arabic) into a common Latin-based encoding before subword tokenization, the authors enable NMT models to better exploit the inherent linguistic similarities between related languages, resulting in translation quality improvements of up to 10 BLEU points.

What Problem Does This Paper Address?

Translating between Indian languages is a "low-resource" challenge. While these languages are spoken by millions, they often lack the massive parallel corpora (paired sentences) required to train high-quality NMT models.

Furthermore, Indian languages are morphologically rich—a single root word can take many forms—and use a variety of scripts. Even though languages like Hindi, Punjabi, and Urdu share deep phonetic and lexical roots, their different writing systems prevent standard NMT models from "seeing" these similarities. A model trained on Devanagari (Hindi) won't naturally recognize the same word written in Perso-Arabic (Urdu), leading to fragmented vocabularies and poor performance in data-scarce scenarios.

Methods

The authors propose a three-step pipeline to create a common "character space" for translation:

  1. WX Encoding (The Core Innovation): Instead of feeding raw scripts into the model, they transliterate both source and target text into WX notation. WX is an ASCII-based transliteration scheme specifically designed for Indian languages. Crucially, it maps phonetically identical sounds to the same Latin characters across different scripts.
  2. Subword Tokenization: They apply Byte Pair Encoding (BPE) via the SentencePiece library on the WX-encoded text. Because the languages now share a common alphabet, BPE can identify shared subwords (roots, prefixes, suffixes) across the language pair, effectively "compressing" the morphological complexity.
  3. Transformer-based NMT: The processed text is used to train a standard Transformer model. After translation, the WX-encoded output is decoded back into the original target script.

The authors also explored Back-translation (generating synthetic parallel data from monolingual text) as an additional way to bolster performance in extremely low-resource settings.

Datasets

The study utilized several prominent repositories for Indian language data:

  • CVIT-PIB: Parallel corpora from the Press Information Bureau.
  • WMT Shared Tasks (2019/2020): Standard benchmarks for machine translation.
  • OPUS & TDIL: Large-scale open-source and government-maintained language repositories.
  • Language Pairs: The primary focus was on Hindi paired with Gujarati, Marathi, Nepali, Maithili, Punjabi, and Urdu.
  • Distant/Zero-Shot: They also tested the approach on Dravidian languages (Malayalam, Tamil, Telugu) and zero-shot pairs (Bhojpuri and Magahi) where no parallel training data was available.

Key Insights and Ideas

  • Phonetics as a Bridge: Most Indian scripts are Brahmi-derived and highly phonetic. By focusing on how words sound (via WX) rather than how they look (via Unicode), the model can leverage "cross-lingual transfer."
  • The Urdu-Hindi Case: Urdu and Hindi are linguistically almost identical in spoken form but use entirely different scripts. WX notation effectively "unifies" them, allowing the model to treat them as the same language with different surface forms.
  • Reducing Surface Complexity: The authors used entropy analysis to show that WX notation, combined with BPE, reduces the "redundancy" of the text. This makes it easier for the neural network to learn the underlying mapping between languages without getting bogged down by script-specific quirks.

Key Results

The proposed WX-based approach consistently outperformed baseline NMT models:

  • Significant Gains: The most dramatic improvement was seen in Nepali $\leftrightarrow$ Hindi translation, with a jump of ~10 BLEU points.
  • Consistent Wins: Most other pairs (Gujarati, Marathi, Punjabi, Urdu) saw improvements ranging from +1 to +5 BLEU points.
  • Zero-Shot Success: The model showed a +1 BLEU point improvement on zero-shot languages like Bhojpuri, proving that the shared phonetic space helps the model generalize to languages it hasn't explicitly seen in training.
  • Back-translation Synergy: Combining WX encoding with back-translation provided an additional boost, particularly for Gujarati $\leftrightarrow$ Hindi.

Limitations and Future Work

  • Script Dependency: The method relies on the availability of high-quality WX converters. If a language lacks a robust transliteration mapping, this approach cannot be easily applied.
  • Distant Language Families: While the method helped with Indo-Aryan to Dravidian translation (e.g., Hindi to Tamil), the improvements were less pronounced than those between closely related Indo-Aryan languages.
  • Future Directions: The authors plan to extend this to a Many-to-Many Multilingual NMT system, where all Indian languages are projected into a single WX-based space. They also aim to incorporate more advanced pre-training techniques like mBART specifically adapted for this shared phonetic-orthographic space.

Summary updated

Abstract

The use of subword embedding has proved to be a major innovation in Neural Machine Translation (NMT). It helps NMT to learn better context vectors for Low Resource Languages (LRLs) so as to predict the target words by better modelling the morphologies of the two languages and also the morphosyntax transfer. Even so, their performance for translation in Indian language to Indian language scenario is still not as good as for resource-rich languages. One reason for this is the relative morphological richness of Indian languages, while another is that most of them fall into the extremely low resource or zero-shot categories. Since most major Indian languages use Indic or Brahmi origin scripts, the text written in them is highly phonetic in nature and phonetically similar in terms of abstract letters and their arrangements. We use these characteristics of Indian languages and their scripts to propose an approach based on common multilingual Latin-based encodings (WX notation) that take advantage of language similarity while addressing the morphological complexity issue in NMT. These multilingual Latin-based encodings in NMT, together with Byte Pair Embedding (BPE) allow us to better exploit their phonetic and orthographic as well as lexical similarities to improve the translation quality by projecting different but similar languages on the same orthographic-phonetic character space. We verify the proposed approach by demonstrating experiments on similar language pairs (Gujarati-Hindi, Marathi-Hindi, Nepali-Hindi, Maithili-Hindi, Punjabi-Hindi, and Urdu-Hindi) under low resource conditions. The proposed approach shows an improvement in a majority of cases, in one case as much as ~10 BLEU points compared to baseline techniques for similar language pairs. We also get up to ~1 BLEU points improvement on distant and zero-shot language pairs.
BibTeX
@misc{kumar2023machine,
  author = {Amit Kumar and Shantipriya Parida and Ajay Pratap and A. K. Singh},
  title = {{Machine Translation by Projecting Text into the Same Phonetic-Orthographic Space Using a Common Encoding}},
  year = {2023},
  eprint = {2305.12371},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2305.12371},
}
Figure