Abstract
South Asia is home to a plethora of languages, many of which severely lack access to new language technologies. This linguistic diversity also results in a research environment conducive to the study of comparative, contact, and historical linguistics—fields which necessitate the gathering of extensive data from many languages. We claim that data scatteredness (rather than scarcity) is the primary obstacle in the development of South Asian language technology, and suggest that the study of language history is uniquely aligned with surmounting this obstacle. We review recent developments in and at the intersection of South Asian NLP and historical-comparative linguistics, describing our and others' current efforts in this area. We also offer new strategies towards breaking the data barrier.
1. Introduction
South Asia is a region of immense linguistic complexity, housing over a billion speakers across at least four major language families: Indo-European, Dravidian, Austroasiatic, and Sino-Tibetan, alongside several language isolates like Burushaski. Despite this richness, the vast majority of these languages are categorized as "low-resource" within the Natural Language Processing (NLP) community. This paper argues that the primary bottleneck for South Asian NLP is not a lack of linguistic data, but rather "data scatteredness." By leveraging the tools of computational historical linguistics, we can unify these disparate resources to build more robust language technologies.
2. The Challenge: Scarcity vs. Scatteredness
The prevailing narrative in NLP suggests that low-resource languages suffer from a fundamental lack of data. However, in the South Asian context, centuries of grammatical descriptions, massive etymological dictionaries, and extensive printed archives exist. The problem is that this data is not "machine-ready." It is often locked in physical books, inconsistent transcription systems, or idiosyncratic PDF formats from old academic theses.
Our research program focuses on "wrangling" this scattered data. We posit that digitizing a 50-year-old grammar book or a century-old linguistic survey can be more effective for model development than attempting to scrape a non-existent web presence for endangered or minority languages.
3. Methodological Framework
To bridge the gap between historical records and modern technology, we employ several interdisciplinary techniques:
- Unified Phonemic Transcription: Comparing languages across different scripts requires a standardized representation. We have developed a system that integrates the International Alphabet of Sanskrit Transliteration (IAST) with the International Phonetic Alphabet (IPA) and Americanist phonetic systems.
- Diachronic Transfer Frameworks: We utilize the evolutionary relationships between language stages (e.g., Old Sinhala and modern Sinhala) to improve dependency parsing. By understanding how a language has changed over time, we can use resources from one historical stage to assist in processing another.
- Computational Reconstruction: We apply supervised and semi-supervised methods to reconstruct "Proto-languages" (ancestral forms), providing a computational basis for studying language isolates and undocumented historical shifts.
- Automated Cognate Extraction: We are consolidating disparate etymological dictionaries into a single, searchable digital format to identify shared word origins across the subcontinent.
4. Key Resource Initiatives
Our efforts have resulted in several critical datasets designed to catalyze South Asian NLP research:
- Jambu: A massive cognate lexicon containing 202,653 lemmas across 294 lects (language varieties). Jambu integrates data from foundational works like the Comparative Dictionary of the Indo-Aryan Languages and the Dravidian Etymological Dictionary.
- Ashokan Prakrit UD Treebank: A new Universal Dependencies (UD) treebank based on 3rd-century BCE inscriptions. This provides a rare computational window into Middle Indo-Aryan syntax.
- UniMorph Expansion: We are expanding the UniMorph database to include complex grammatical paradigms for more South Asian languages, with a specific focus on those with intricate tonal systems.
- Linguistic Survey of India (LSI) Digitization: We are applying modern OCR and NLP extraction techniques to the century-old, multi-volume LSI to unlock its vast store of linguistic data.
5. Insights and Results
Our research demonstrates that historical linguistics is a powerful tool for modern technology. Because South Asian languages have influenced each other for millennia—forming a "linguistic area"—etymological data can be used for transfer learning. Models trained on high-resource languages can be mapped onto low-resource relatives using cognate data from Jambu.
Furthermore, our work has successfully mapped tonogenesis (the development of tonal contrasts) in the languages of Northern Pakistan, such as Gawri and Torwali. We are also investigating "unattested substrates"—extinct languages that left traces in modern vocabularies—which may help solve long-standing mysteries regarding the prehistory of the region, including the language of the Indus Valley Civilization.
6. Limitations and Future Directions
Significant challenges remain, most notably the OCR bottleneck. Performing Optical Character Recognition on old texts with complex diacritics remains difficult. Additionally, language isolates like Burushaski pose a unique challenge because they lack relatives to provide transfer data.
Moving forward, we emphasize that digital archives must be supplemented by fieldwork. We are launching new initiatives to elicit data from endangered languages like Kholosi and Huvadhoo to ensure these communities are included in the development of future language technologies.
7. Conclusion
By reframing the problem of South Asian languages from one of scarcity to one of scatteredness, we open new pathways for resource development. Computational historical linguistics provides the necessary bridge to transform centuries of linguistic scholarship into the building blocks of modern NLP, ensuring that the region's linguistic diversity is preserved and empowered in the digital age.
Summary updated