South Asia is home to over a quarter of the global population and a vast array of languages spanning at least four major families. Despite this richness, the majority of these languages are classified as "low-resource" within the Natural Language Processing (NLP) community. A study presented at ACL 2022 challenges this classification, suggesting that the primary obstacle is not a lack of data, but rather its "scatteredness"—the fact that it remains trapped in non-digital formats or isolated linguistic silos.
The Paradigm of Data Scatteredness
The authors argue that for many South Asian languages, substantial linguistic documentation exists in the form of historical surveys, dictionaries, and printed texts. However, this information is not "machine-ready." The bottleneck for developing NLP tools like machine translation or spellcheckers is therefore an engineering challenge of digitization and standardization rather than a fundamental lack of information.
Methodological Framework
To address this "scatteredness," the research proposes an interdisciplinary approach leveraging computational historical linguistics:
- Digitization and OCR: Utilizing neural models (such as Transkribus) to convert historical linguistic records and surveys into structured, searchable data.
- Dependency Treebanking: Developing syntactically annotated corpora under the Universal Dependencies (UD) framework. This facilitates "cross-lingual transfer," where models trained on high-resource languages like Hindi can be adapted to related low-resource languages like Bhojpuri or Magahi.
- Etymological Consolidation: Building large-scale databases of cognates (words with common origins) to help models understand inter-language relationships and improve multilingual tasks.
- Unified Transcription: Implementing a consistent phonemic transcription system that bridges different language families by combining IAST and IPA standards.
Key Datasets and Resources
A cornerstone of this effort is Jambu, a massive etymological database containing over 202,653 lemmas across 294 language varieties. It consolidates data from foundational sources like Turner’s Comparative Dictionary of the Indo-Aryan Languages, providing a digital foundation for comparative linguistics.
Other significant contributions include:
- Ashokan Prakrit UD Treebank: A parallel corpus of 3rd-century BCE inscriptions, representing the first computational study of Middle Indo-Aryan regional fragmentation.
- UniMorph Contributions: Morphological paradigm data for several South Asian languages, specifically addressing languages that signal grammatical changes through tone.
- LSI Digitization: Ongoing efforts to digitize the Linguistic Survey of India, a century-old survey that remains the most comprehensive source for many minor languages.
Insights and Future Directions
The study emphasizes that historical data serves as a "genetic" blueprint for modern languages. By shifting focus to the digitization of existing records, researchers can unlock resources for "Level 0" (left-behind) languages. While challenges remain—such as diacritic recognition in OCR and the need for fieldwork for truly data-scarce isolates like Kusunda—this roadmap provides a clear path toward integrating South Asia's linguistic diversity into the global NLP ecosystem.
Summary updated