A House United: Bridging the Script and Lexical Barrier between Hindi and Urdu

Riyaz A. Bhat, Irshad A. Bhat, Naman Jain, Dipti Misra SharmaCOLING 2016

TL;DR

This paper tackles the challenge of treating Hindi and Urdu as separate entities in NLP, despite their shared linguistic roots. They propose a unified approach by harmonizing their scripts (transliterating Urdu to Devanagari) and bridging lexical differences using cross-register word embeddings, leading to improved performance in dependency parsing and enabling interchangeable use of models.

What Problem Does This Paper Address?

Hindi and Urdu, while linguistically very similar and often considered dialects of a single language (Hindustani), are treated as distinct languages in computational linguistics. This separation stems from their different writing systems (Devanagari for Hindi, Perso-Arabic for Urdu) and divergent literary vocabularies. Consequently, separate NLP models and resources are developed for each, leading to duplicated effort and suboptimal performance due to data sparsity. The paper aims to bridge this gap by creating a unified processing framework that leverages the shared linguistic foundation of Hindi and Urdu.

Methods

The authors propose a two-pronged approach to harmonize Hindi and Urdu texts:

  • Script Harmonization: They address the orthographic barrier by learning accurate statistical transliteration models. Crucially, they employ sentence-level decoding with n-best transliterations to resolve word ambiguity that arises from the Perso-Arabic script's lack of explicit short vowels. They empirically demonstrate that using the Devanagari script for both languages leads to better performance in downstream NLP tasks.
  • Lexical Harmonization: To overcome lexical divergences, they learn cross-register word embeddings. This involves training word embeddings on a combined corpus of transliterated Hindi and Urdu texts, effectively projecting words with similar meanings into a shared semantic space, regardless of their original script or origin (Sanskrit vs. Persian/Arabic).

These harmonized representations are then used to train neural network-based dependency parsers, exploring resource sharing and augmentation strategies.

Datasets

The experiments utilize existing treebanks for Hindi and Urdu:

  • Hindi and Urdu Treebanks (HDTB and UDTB): These multi-layered treebanks provide annotations for dependency structure, PropBank-style argument structure, and phrase structure. For this work, only the dependency structure layer is used.
  • Hindi and Urdu Monolingual Corpora: Large corpora are used for learning word embeddings.
  • ILCI Hindi-Urdu Parallel Corpora: Used to extract transliteration pairs.
  • IndoWordNet: Also used to extract transliteration pairs.

No new benchmarks are introduced, but the paper evaluates performance on existing Hindi and Urdu treebank data.

Key Insights and Ideas

  • Devanagari as a Unifying Script: The paper strongly advocates for using Devanagari as the common representation for both Hindi and Urdu in computational linguistics. This choice significantly improves performance across various NLP modules (POS tagging, chunking, dependency parsing) compared to using the Perso-Arabic script.
  • Transliteration for Ambiguity Resolution: Simply transliterating isn't enough; the ability to resolve word ambiguity (e.g., distinguishing between "air" and "became" in Urdu due to missing vowels) is critical for effective processing. Their noisy-channel model for sentence-level decoding proves effective here.
  • Lexical Divergence is the Bigger Hurdle: While script differences are significant, the paper highlights that lexical differences are the primary reason for Hindi and Urdu being unintelligible. Bridging these gaps through shared word embeddings is crucial.
  • Resource Augmentation is Key: Combining training data from both languages and using techniques like LININT and WEIGHTED methods for augmentation leads to substantial improvements, especially for Urdu, which often has smaller treebanks. This suggests Hindi and Urdu resources are highly complementary.
  • Interchangeable Models: The harmonized approach allows for models trained on one language to perform reasonably well on the other, demonstrating the effectiveness of bridging the script and lexical barriers.

Key Results

  • Transliteration Accuracy: Their Devanagari to Perso-Arabic transliteration system achieves 98.03% accuracy, and Perso-Arabic to Devanagari reaches 88.03% (with a noisy-channel model improving this to 96.72% on the test set).
  • Extrinsic Evaluation: Using Devanagari for both Hindi and Urdu consistently yields higher accuracies across POS tagging, chunking, and dependency parsing compared to using Perso-Arabic. For instance, dependency parsing accuracy with gold features is 91.70% for Hindi in Devanagari vs. 88.08% for Urdu in Perso-Arabic.
  • Resource Augmentation: Domain adaptation methods, particularly LININT, significantly improve performance. For dependency parsing, accuracies increase from ~79% (Hindi) and ~75% (Urdu) for SRCONLY (resource sharing) to ~82% (Hindi) and ~88% (Urdu) with augmentation methods.
  • Interchangeability: Parsers trained on harmonized Hindi and Urdu resources can be used interchangeably, achieving near state-of-the-art results.

Limitations and Future Work

  • Limitations: The paper acknowledges that while Devanagari is preferred, the Perso-Arabic to Devanagari transliteration still struggles with missing vowels, impacting accuracy. The performance of the SRCONLY models (resource sharing without augmentation) is not as strong as expected, likely due to domain shift issues.
  • Future Work: The authors suggest exploring the merging of semantic role annotations for better semantic role labeling and investigating whether their findings on resource sharing and augmentation hold for applications beyond parsing.

Summary updated

BibTeX
@inproceedings{bhat2016house,
  author = {Riyaz A. Bhat and Irshad A. Bhat and Naman Jain and Dipti Misra Sharma},
  title = {{A House United: Bridging the Script and Lexical Barrier between Hindi and Urdu}},
  booktitle = {International Conference on Computational Linguistics},
  year = {2016},
}
Figure