How do different factors Impact the Inter-language Similarity? A Case Study on Indian languages

Sourav Kumar, Salil Aggarwal, Dipti Misra Sharma, Radhika MamidiACL 2021

TL;DR

This paper investigates how various factors, such as algorithms, scripts, and dataset size, influence the lexical similarity calculations between Indian languages. The authors found that while specific similarity scores vary, the relative order of language relatedness remains remarkably consistent, suggesting that language families and geographical proximity are strong predictors of lexical similarity.

What Problem Does This Paper Address?

Natural Language Processing (NLP) systems often struggle with low-resource languages due to the need for extensive linguistic knowledge or large amounts of labeled data. Indian languages, despite their rich cultural heritage, present a significant challenge due to their sheer diversity and the complexity of their morphology. Exploiting the inherent relatedness between these languages is crucial for improving NLP tasks like machine translation and sentiment analysis. However, a comprehensive study on how to best measure this relatedness, and what factors influence these measurements, has been lacking. This paper aims to fill that gap by systematically analyzing the impact of different methodologies and data characteristics on inter-language similarity calculations for Indian languages.

Methods

The authors propose a pipeline for calculating inter-language similarity that involves:

  1. Collecting parallel data: Using sentence-aligned corpora for language pairs.
  2. Unified Transliteration: Mapping all languages to a common script (Devanagari or Abugida) to enable direct comparison.
  3. Sentence-to-Sentence Similarity Calculation: Employing several standard string similarity algorithms:
  4. Token Overlap: Measures the proportion of shared words (tokens) between sentences.
  5. Levenshtein Distance: Calculates the minimum number of single-character edits (insertions, deletions, substitutions) required to transform one string into another.
  6. Longest Common Subsequence (LCS): Finds the longest sequence of characters common to both strings, not necessarily contiguous.
  7. Shingle (q-gram) Similarity: Compares strings based on the similarity of their overlapping character n-grams (shingles).

The study also explores similarity at both the word and sub-word (character n-gram) levels.

Datasets

The research utilizes two primary datasets:

  • ILCI (Indian Language Corpora Initiative): Contains 50,000 sentences across major Indian languages (Hindi, Urdu, Punjabi, Gujarati, Marathi, Bangla, Konkani, Telugu, Tamil, Malayalam) covering health and tourism domains.
  • PMI (Prime Minister of India): A collection of 30,000 sentences from news domains for most of the languages in ILCI, plus Oriya and Kannada, excluding Konkani.

Key Insights and Ideas

  • Robustness of Relative Similarity: The most striking finding is that the relative order of similarity between language pairs remains largely consistent across different similarity algorithms. This suggests that inherent linguistic relationships (like language family and geographical proximity) are stronger predictors of similarity than the specific algorithm used.
  • Language Family Matters: Similarity is significantly higher within the same language family (Indo-Aryan or Dravidian).
  • Geographical Proximity Enhances Similarity: Even across language families, languages that are geographically close tend to show higher similarity (e.g., Telugu and Marathi).
  • Sub-word Level Captures More: Similarity is more pronounced at the sub-word (character n-gram) level compared to the word level, indicating that shared roots and morphological similarities are better captured by looking at smaller character sequences.
  • Data Size Threshold: A relatively small parallel dataset (around 2,000 sentences) appears sufficient for the similarity score to stabilize, suggesting that extensive data might not always be necessary for establishing relative language relatedness.
  • Script Independence: The choice of common script for transliteration (Devanagari vs. Abugida) has a negligible impact on the calculated similarity scores.

Key Results

  • Most Similar Pair: Hindi and Urdu consistently emerge as the most similar language pair across all experiments.
  • Least Similar Pair: Tamil and Punjabi are generally found to be among the least similar.
  • Algorithm Variation: While the relative order of similarity is stable, absolute similarity scores do vary between algorithms. For instance, Token Overlap and Levenshtein Distance tend to show higher similarity scores than LCS or Shingle similarity.
  • Dataset Domain Insignificant: The domain of the dataset (health/tourism vs. news) does not significantly alter the relative similarity matrix, indicating that lexical similarity is relatively domain-independent.
  • Root Word vs. Full Word: Considering only root words (ignoring affixes) drastically increases the observed similarity, highlighting the importance of morphological richness in Indian languages.

Limitations and Future Work

  • Lexical Focus: The study primarily focuses on lexical similarity. The authors suggest that incorporating semantic similarity alongside lexical similarity could further enhance accuracy and provide a more comprehensive understanding of language relatedness.
  • Morphological Richness: While acknowledged, the full impact of the complex morphology of Indian languages on similarity calculations could be explored further.
  • Future Research: Future work could extend this study to include semantic similarity, explore more sophisticated NLP techniques for similarity measurement, and investigate the impact of other factors like phonological similarity.

Summary updated

BibTeX
@inproceedings{kumar2021how,
  author = {Sourav Kumar and Salil Aggarwal and Dipti Misra Sharma and Radhika Mamidi},
  title = {{How do different factors Impact the Inter-language Similarity? A Case Study on Indian languages}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2021},
}
Figure