Low-Resource Transliteration for Roman-Urdu and Urdu Using Transformer-Based Models

Umer Butt, Stalin Varanasi, Günter NeumannNAACL 2025

TL;DR

This paper tackles the challenge of accurately transliterating between Roman-Urdu and standard Urdu, two scripts widely used but with significant spelling variations in Roman-Urdu. They leverage a powerful multilingual transformer model (m2m100), enhanced with masked language modeling (MLM) pretraining and careful dataset splitting, to achieve state-of-the-art performance and better generalization across different data sources.

What Problem Does This Paper Address?

The digital age has seen a surge in the use of Roman-Urdu, a Latin-alphabet representation of Urdu, especially on social media and messaging platforms. However, this script lacks standardized spelling, leading to high variability and making it difficult for NLP models to accurately process. This poses a significant challenge for tasks like machine translation and information retrieval, especially for low-resource languages like Urdu. While previous work has explored RNN-based approaches, they often struggle with domain adaptability and suffer from inconsistent evaluation methodologies, leading to unreliable performance metrics. This paper aims to bridge this gap by developing a robust transliteration system that can handle the nuances of Roman-Urdu and generalize well across different datasets.

Methods

The core of the proposed solution is the m2m100 multilingual translation model, a powerful transformer-based architecture. The authors enhance this model in several key ways:

  • Masked Language Modeling (MLM) Pretraining: Before fine-tuning for transliteration, the model undergoes MLM pretraining on monolingual Roman-Urdu and Urdu text. This helps the model internalize script-specific variations and subword structures, improving its ability to handle inconsistent spellings and enhancing generalization. They experimented with two MLM configurations: one trained only on Roman-Urdu, and another on both Roman-Urdu and Urdu.
  • Fine-tuning for Transliteration: After MLM pretraining, the model is fine-tuned on parallel Roman-Urdu and Urdu data. They use the Roman-Urdu-Parl (RUP) dataset for initial fine-tuning and then further fine-tune on the Dakshina dataset to improve domain adaptation.
  • Careful Dataset Splitting: A critical contribution is the correction of the RUP dataset's evaluation methodology. They implemented a structured train-test split to prevent data leakage, ensuring that no Urdu sentence or its variations appeared in multiple subsets. This rigorous splitting strategy aims to provide a more accurate and reliable evaluation of the model's generalization capabilities.
  • Language-Specific Tokens: To explicitly condition the model on the source and target scripts, they introduced language-specific tokens (__roman-ur__ and __ur__) to the m2m100 tokenizer.

Datasets

The paper utilizes two primary datasets:

  • Roman-Urdu-Parl (RUP): A large-scale dataset containing approximately 6.3 million sentence pairs of Roman-Urdu and Urdu, collected from informal online sources. This serves as the primary dataset for training and evaluation.
  • Dakshina Dataset: A smaller, domain-specific dataset of 10,000 parallel sentences sourced from Google's Dakshina dataset, which includes Roman-Urdu text from Wikipedia. This dataset is used to assess domain generalization.

The authors also highlight issues with the original RUP dataset's test set construction and propose their own structured split to address data leakage.

Key Insights and Ideas

  • MLM Pretraining is Key for Generalization: The most significant insight is that MLM pretraining, even on raw text, substantially improves the model's ability to generalize across different datasets (RUP and Dakshina) without significantly harming performance on the primary training dataset. This suggests that understanding script-level variations independently of direct transliteration mappings is crucial.
  • Transformer Models Outperform RNNs: The results clearly demonstrate that transformer-based models, particularly when fine-tuned with MLM pretraining, significantly outperform traditional RNN+LSTM approaches for Roman-Urdu transliteration.
  • The Challenge of Roman-Urdu Standardization: The paper underscores the inherent ambiguity in Urdu-to-Roman-Urdu transliteration due to the lack of standardized spelling in Roman-Urdu. This makes it a more challenging task than Roman-Urdu-to-Urdu transliteration.
  • Rigorous Evaluation is Crucial: The authors emphasize the importance of proper dataset splitting and character-level metrics (Char-BLEU, CHRF) for accurately evaluating transliteration tasks, as n-gram BLEU can be misleading.

Key Results

  • State-of-the-Art Performance: The proposed MLM-pretrained m2m100 model achieves state-of-the-art performance, with Char-BLEU scores of 96.37% for Urdu → Roman-Urdu and 97.44% for Roman-Urdu → Urdu.
  • Improved Generalization: The MLM-pretrained models show more balanced performance across datasets, mitigating the trade-off between domain adaptation and retention of previously learned transliteration patterns.
  • Outperforming Baselines: The model outperforms both previous RNN+LSTM baselines and the zero-shot GPT-40 Mini model on the RUP dataset. For example, the RNN+LSTM model achieved a BLEU score of 84.67, while their model achieved 94.58.
  • GPT-40 Mini Performance: OpenAI's GPT-40 Mini, while a strong multilingual baseline, performed competitively on Dakshina but lagged behind the fine-tuned model on RUP, especially for Urdu → Roman-Urdu.

Limitations and Future Work

  • Ambiguity in Roman-Urdu: The inherent lack of standardization in Roman-Urdu spelling remains a challenge. Even with character-level metrics, plausible variants not present in the reference can be penalized.
  • Multiple Reference Evaluation: The authors suggest exploring multi-reference evaluation or designing more flexible scoring metrics to better handle the variability in Roman-Urdu.
  • MLM for Other Low-Resource Scripts: They propose that MLM pretraining could be beneficial for other non-standardized Romanized scripts where dedicated training data is scarce.
  • Semi-Supervised Learning: Investigating semi-supervised learning approaches that leverage unlabeled Roman-Urdu text is suggested as a way to reduce reliance on parallel corpora.

Summary updated

BibTeX
@inproceedings{butt2025low,
  author = {Umer Butt and Stalin Varanasi and Günter Neumann},
  title = {{Low-Resource Transliteration for Roman-Urdu and Urdu Using Transformer-Based Models}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2025},
}
Figure