Abstract
We introduce Jambu, a cognate database of South Asian languages which unifies dozens of previous sources in a structured and accessible format. The database includes 287k lemmata from 602 lects, grouped together in 23k sets of cognates. We outline the data wrangling necessary to compile the dataset and train neural models for reflex prediction on the Indo-Aryan subset of the data. We hope that Jambu is an invaluable resource for all historical linguists and Indologists, and look towards further improvement and expansion of the database.
Introduction
Historical linguistics seeks to understand the evolution of languages by comparing cognates—words across different languages that share a common ancestor. South Asia is one of the most linguistically diverse regions on Earth, serving as home to the Indo-Aryan, Dravidian, Austroasiatic (Munda), and Tibeto-Burman language families. Despite this richness, the region has remained computationally understudied.
Historically, high-quality etymological data was confined to massive, non-standardized print dictionaries compiled in the mid-20th century. Because these sources utilized disparate transcription schemes and lacked a unified digital structure, researchers were unable to easily apply modern computational methods, such as automatic ancestral reconstruction or reflex prediction, to South Asian linguistic data. Jambu addresses this gap by digitizing and reconciling decades of scholarship into a single, standardized open-access database.
The Jambu Database
Jambu represents a massive consolidation of linguistic knowledge, containing:
- 287,135 lemmata (individual word forms).
- 602 lects (distinct languages and geographical dialects).
- 23,024 cognate sets.
The database integrates classic works, such as Turner’s Comparative Dictionary of the Indo-Aryan Languages (CDIAL) and Burrow & Emeneau’s Dravidian Etymological Dictionary (DEDR), with dozens of modern sources. Notably, it includes fieldwork on endangered or under-documented languages like Kholosi and Shina, providing a granular look at the region's linguistic landscape.
Methodology
The construction of Jambu involved a rigorous four-stage data-engineering pipeline:
- Scraping and Parsing: Data was extracted from digital versions of foundational etymological dictionaries.
- Standardization: The authors adopted the Cross-Linguistic Data Format (CLDF), a modern standard that facilitates the representation of complex relationships, including loanwords and phonetic alignments.
- Orthographic Normalization: To resolve the disparate transcription systems used across different sources, a rigid transcription system based on Indological conventions was developed. Python-based automation converted over 200,000 entries into this standardized format with 99.7% accuracy.
- Neural Modeling: To validate the dataset's utility for machine learning, the authors trained Gated Recurrent Unit (GRU) and Transformer architectures on the task of reflex prediction—predicting a modern word form based on its ancient ancestor.
Key Insights and Results
Jambu highlights the "linguistic mesh" of South Asia, where language families have influenced one another's grammar and vocabulary for millennia. By including 32 different geographical dialects for languages like Shina, the database captures fine-grained language contact that broader datasets often overlook.
The neural modeling experiments established baseline performances for reflex prediction on the Indo-Aryan subset. The Transformer model achieved a BLEU score of 56.03 and a Translation Edit Rate (TER) of 35.15, while the GRU model achieved a BLEU of 55.91 and a TER of 34.40. These results demonstrate that the models can effectively "learn" the regular sound changes described by the Neogrammarian hypothesis, proving that Jambu is of sufficient quality for complex downstream Natural Language Processing (NLP) tasks.
Limitations and Future Work
While Jambu is a significant step forward, several challenges remain. The current iteration does not yet perfectly distinguish between inherited words and loanwords (borrowings). Additionally, many languages in the Northwest (such as Punjabi) are tonal, but older dictionaries often failed to record these tones. Future versions of Jambu will aim to integrate modern fieldwork to fill these gaps and continue the manual expert review necessary to correct errors inherent in the original historical sources. By turning a messy human-curated archive into a machine-learnable dataset, Jambu provides a critical foundation for the future of computational historical linguistics in South Asia.
Summary updated