Jambu: A historical linguistic database for South Asian languages

Aryaman Arora, Adam Farris, Samopriya Basu, Suresh Kolichala

TL;DR

This paper introduces JAMBU, the largest and most comprehensive historical linguistic database for South Asian languages to date. By unifying data from numerous sources into a structured format, JAMBU aims to facilitate computational research into language evolution and contact in the region, a field previously hampered by fragmented and inaccessible data.

What Problem Does This Paper Address?

Understanding how languages evolve and interact over time is a core challenge in historical linguistics. This often involves identifying "cognates" – words that share a common ancestor and have undergone regular sound changes. While computational methods are increasingly applied to this field, progress in South Asia has been significantly hindered by the lack of a unified, accessible database of cognates. Existing resources are often scattered, difficult to parse, and not updated, making large-scale computational analysis challenging. This paper addresses this gap by creating JAMBU, a structured database designed to support computational research on South Asian historical linguistics and language contact.

Methods

The JAMBU database was constructed by aggregating data from various sources, primarily the comprehensive etymological dictionaries CDIAL (Turner, 1962–1966) and DEDR (Burrow and Emeneau, 1984). The data was wrangled and structured using the CLDF (Cross-Linguistic Data Formats) standard, which is designed to be flexible for linguistic data. A significant methodological challenge was reconciling diverse transcription systems from different sources into a unified, standardized transcription scheme based on Indological conventions. For evaluation, the authors replicated a reflex prediction task, training neural models (a bidirectional GRU encoder-decoder and a transformer encoder-decoder) on the Indo-Aryan subset of JAMBU to predict the descendant form of a word in a given Indo-Aryan language.

Datasets

The primary dataset introduced is JAMBU, a cognate database for South Asian languages. It contains approximately 287,000 lemmata from 602 lects, grouped into 23,000 cognate sets. The data is drawn from major etymological dictionaries for Indo-Aryan and Dravidian languages, as well as other recent sources. For the experimental evaluation, the Indo-Aryan subset of JAMBU was used, with 80% of the data for training and 20% for testing.

Key Insights and Ideas

  • The critical need for unified data: The paper highlights how the fragmentation of linguistic data is a major bottleneck for computational historical linguistics, especially in complex regions like South Asia. JAMBU's creation is a direct response to this.
  • Reconciling transcription diversity is a major hurdle: The authors emphasize the difficulty of standardizing phonetic and orthographic representations from various sources, which is crucial for computational analysis.
  • A broad definition of "cognate" is useful: The paper notes that while historical linguists often focus on strict etymological sources, computational work can benefit from a broader definition that might include loanwords, enabling more comprehensive analyses of language contact.
  • Computational methods can leverage existing, but previously inaccessible, data: By creating JAMBU, the authors demonstrate how modern NLP techniques can be applied to vast amounts of historical linguistic data that were previously too difficult to process.

Key Results

The JAMBU database contains 287,135 lemmata from 602 lects, organized into 23,024 cognate sets, covering Indo-Aryan, Dravidian, Munda, and Nuristani language families. The authors developed a standardized transcription system that successfully processed 99.7% of 204,000 lemmata automatically. In their reflex prediction experiment on the Indo-Aryan subset, both the GRU and Transformer models achieved comparable performance, with perplexity scores around 2.5 and BLEU scores around 56.

Limitations and Future Work

The authors acknowledge that many sources are yet to be incorporated into JAMBU, particularly those detailing loanwords from external languages and local literary traditions. Future work will focus on disentangling cross-lectal interactions and marking lexical isoglosses to better understand language contact history. They also plan to extend the database structure to support more complex cross-lingual interactions, such as distinguishing inheritance from loaning. Additionally, they aim to incorporate consistent orthography for tonemes in tonal languages and manually improve data quality by fixing errors and reorganizing entries.

Summary updated

Abstract

We introduce Jambu, a cognate database of South Asian languages which unifies dozens of previous sources in a structured and accessible format. The database includes 287k lemmata from 602 lects, grouped together in 23k sets of cognates. We outline the data wrangling necessary to compile the dataset and train neural models for reflex prediction on the Indo-Aryan subset of the data. We hope that Jambu is an invaluable resource for all historical linguists and Indologists, and look towards further improvement and expansion of the database.
BibTeX
@inproceedings{arora2023jambu,
  author = {Aryaman Arora and Adam Farris and Samopriya Basu and Suresh Kolichala},
  title = {{Jambu: A historical linguistic database for South Asian languages}},
  booktitle = {SIGMORPHON},
  year = {2023},
  url = {https://openreview.net/forum?id=jTCA91JYz},
}
Figure