Computational Historical Linguistics and Language Diversity in South Asia

Aryaman Arora, Adam Farris, Samopriya Basu, Suresh Kolichala

South Asia is home to over a quarter of the global population and a vast array of languages spanning at least four major families. Despite this richness, the majority of these languages are classified as "low-resource" within the Natural Language Processing (NLP) community. A study presented at ACL 2022 challenges this classification, suggesting that the primary obstacle is not a lack of data, but rather its "scatteredness"—the fact that it remains trapped in non-digital formats or isolated linguistic silos.

The Paradigm of Data Scatteredness

The authors argue that for many South Asian languages, substantial linguistic documentation exists in the form of historical surveys, dictionaries, and printed texts. However, this information is not "machine-ready." The bottleneck for developing NLP tools like machine translation or spellcheckers is therefore an engineering challenge of digitization and standardization rather than a fundamental lack of information.

UD Corpus Size by Language illustrates the significant disparity in digital resources across South Asian languages.

Methodological Framework

To address this "scatteredness," the research proposes an interdisciplinary approach leveraging computational historical linguistics:

  • Digitization and OCR: Utilizing neural models (such as Transkribus) to convert historical linguistic records and surveys into structured, searchable data.
  • Dependency Treebanking: Developing syntactically annotated corpora under the Universal Dependencies (UD) framework. This facilitates "cross-lingual transfer," where models trained on high-resource languages like Hindi can be adapted to related low-resource languages like Bhojpuri or Magahi.
  • Etymological Consolidation: Building large-scale databases of cognates (words with common origins) to help models understand inter-language relationships and improve multilingual tasks.
  • Unified Transcription: Implementing a consistent phonemic transcription system that bridges different language families by combining IAST and IPA standards.

Key Datasets and Resources

A cornerstone of this effort is Jambu, a massive etymological database containing over 202,653 lemmas across 294 language varieties. It consolidates data from foundational sources like Turner’s Comparative Dictionary of the Indo-Aryan Languages, providing a digital foundation for comparative linguistics.

Distribution of lemmas across languages in the Jambu database, highlighting the depth of historical data available.

Other significant contributions include:

  • Ashokan Prakrit UD Treebank: A parallel corpus of 3rd-century BCE inscriptions, representing the first computational study of Middle Indo-Aryan regional fragmentation.
  • UniMorph Contributions: Morphological paradigm data for several South Asian languages, specifically addressing languages that signal grammatical changes through tone.
  • LSI Digitization: Ongoing efforts to digitize the Linguistic Survey of India, a century-old survey that remains the most comprehensive source for many minor languages.

Insights and Future Directions

The study emphasizes that historical data serves as a "genetic" blueprint for modern languages. By shifting focus to the digitization of existing records, researchers can unlock resources for "Level 0" (left-behind) languages. While challenges remain—such as diacritic recognition in OCR and the need for fieldwork for truly data-scarce isolates like Kusunda—this roadmap provides a clear path toward integrating South Asia's linguistic diversity into the global NLP ecosystem.

Summary updated

Abstract

South Asia is home to a plethora of languages, many of which severely lack access to new language technologies. This linguistic diversity also results in a research environment conducive to the study of comparative, contact, and historical linguistics -- fields which necessitate the gathering of extensive data from many languages. We claim that data scatteredness (rather than scarcity) is the primary obstacle in the development of South Asian language technology, and suggest that the study of language history is uniquely aligned with surmounting this obstacle. We review recent developments in and at the intersection of South Asian NLP and historical-comparative linguistics, describing our and others' current efforts in this area. We also offer new strategies towards breaking the data barrier.
BibTeX
@inproceedings{arora2022computational,
  author = {Aryaman Arora and Adam Farris and Samopriya Basu and Suresh Kolichala},
  title = {{Computational Historical Linguistics and Language Diversity in South Asia}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2022},
  url = {https://openreview.net/forum?id=KIWhhdbp2OG},
}
Figure