Harnessing Cross-lingual Features to Improve Cognate Detection for Low-resource Languages

Diptesh Kanojia, Raj Dabre, Shubham Dewangan, Pushpak Bhattacharyya, Gholamreza Haffari, Malhar Kulkarni

Abstract

Cognate detection—identifying words with shared etymological origins—is a foundational task in Natural Language Processing (NLP). Traditional methods relying on orthographic or phonetic similarity often fail for low-resource languages with diverse scripts or significant morphological divergence. This paper presents a novel approach for 14 Indian languages by integrating cross-lingual word embeddings with semantic context from lexical knowledge graphs. Our hybrid model, combining MUSE embeddings with Weighted Lexical Similarity, achieves up to an 18% improvement in F-score over state-of-the-art baselines. Furthermore, we demonstrate that incorporating these detected cognates into Neural Machine Translation (NMT) pipelines yields gains of up to 2.76 BLEU points.

Introduction

In linguistically dense regions like the Indian subcontinent, languages from different families (e.g., Indo-Aryan and Dravidian) often share cognates that are difficult to detect due to varying scripts and morphological changes. For instance, the Hindi word Aag and the Telugu word Agni are cognates but lack obvious orthographic similarity in their native scripts. This research addresses whether semantic information from cross-lingual models can bridge this gap for low-resource languages.

Methodology

The proposed detection framework utilizes a multi-stage pipeline to integrate lexical, phonetic, and semantic features.

The multi-stage cognate detection pipeline utilizing lexical, phonetic, and semantic features.
  1. Transliteration: Languages are converted to a common Devanagari script to mitigate multi-script issues.
  2. Context Augmentation: Using IndoWordNet, the authors extract definitions and example sentences to provide a semantic footprint that simple word lists lack.
  3. Cross-lingual Embeddings: Three methodologies—MUSE (supervised), VecMap (robust mapping), and XLM-R (transformer-based)—are compared for mapping languages into a shared mathematical space.
  4. Classification: A Feed-Forward Neural Network (FFNN) classifies word pairs based on features including Weighted Lexical Similarity (WLS), phonetic vectors, and angular cosine similarity of the embeddings.

Datasets and Experimental Setup

The study expanded an existing challenge dataset of 12 Indian languages to include Konkani and Nepali. Monolingual corpora were sourced from Wikimedia, the Indian Language Corpora Initiative (ILCI), and web crawls, while the ILCI Phase 1 corpus provided parallel data for downstream NMT evaluation. Semantic context was derived from linked Indian language wordnets.

Results and Discussion

The experimental results demonstrate that semantic similarity and context significantly outperform orthographic baselines, particularly for morphologically rich Dravidian languages like Tamil and Telugu.

Performance comparison of baseline and cross-lingual embedding-based approaches across 13 language pairs.

As shown in the results table, the hybrid "MUSE + WLS w/ FFNN" model consistently provided the best performance. For the Hindi-Tamil (Hi-Ta) pair, the method achieved an 18% point improvement in F-score over previous baselines. Interestingly, simpler models like MUSE and VecMap outperformed the complex XLM-R, likely due to the limited data available for these low-resource languages. In downstream tasks, injecting detected cognates into NMT training improved translation quality by up to 2.76 BLEU for Hindi-to-Punjabi.

Conclusion

This work highlights the power of hybrid features—combining semantic signals with traditional lexical anchors—for cognate detection in low-resource settings. While challenges remain regarding agglutinative languages and transformer performance on small datasets, the release of these models and datasets provides a strong foundation for future research in multilingual NLP for the Indian subcontinent.

Summary updated

BibTeX
@inproceedings{kanojia2020harnessing,
  author = {Diptesh Kanojia and Raj Dabre and Shubham Dewangan and Pushpak Bhattacharyya and Gholamreza Haffari and Malhar Kulkarni},
  title = {{Harnessing Cross-lingual Features to Improve Cognate Detection for Low-resource Languages}},
  booktitle = {International Conference on Computational Linguistics},
  year = {2020},
  doi = {10.18653/v1/2020.coling-main.119},
}
Figure