Abstract
Tamil, a Dravidian language of South Asia, is a highly diglossic language with two very different registers in everyday use: Literary Tamil (preferred in writing and formal communication) and Spoken Tamil (confined to speech and informal media). Spoken Tamil is under-supported in modern NLP systems. In this paper, we release IruMozhi, a human-annotated dataset of parallel text in Literary and Spoken Tamil. We train classifiers on the task of identifying which variety a text belongs to. We use these models to gauge the availability of pretraining data in Spoken Tamil, to audit the composition of existing labelled datasets for Tamil, and to encourage future work on the variety.
Introduction
Tamil is a diglossic language, meaning it exists in two distinct forms: a "High" variety (Literary Tamil) used for formal writing, news, and education, and a "Low" variety (Spoken Tamil) used in daily conversation and social media. While most users prefer to interact with technology using the Spoken variety, almost all existing Natural Language Processing (NLP) tools and training data are focused on Literary Tamil. This creates a significant "representation gap" where AI models struggle to understand the informal, colloquial language used by millions. Furthermore, Spoken Tamil lacks a standardized spelling system and is frequently written using the Latin alphabet (Romanization) online, making it even harder for traditional models to process.
The IruMozhi Dataset
To bridge this gap, we introduced IruMozhi, a human-annotated parallel dataset. We began with 499 Literary Tamil sentences sourced from Wikipedia, transliterated them into the Latin script, and had two native speakers "translate" them into Spoken Tamil.
Because Spoken Tamil has no fixed spelling, we implemented rule-based data augmentation to generate orthographic variants. This process expanded the dataset from approximately 1,500 lines to over 8,600 lines, making the resulting models more robust to spelling variations found in the wild.
Methods and Classification
We trained two types of models to identify the linguistic register:
- Naïve Bayes: A classic statistical approach using character and word n-grams.
- XLM-R: A state-of-the-art transformer-based model pretrained on over 100 languages.
These models were used to "audit" existing Tamil datasets to estimate the prevalence of Spoken vs. Literary content. Evaluated datasets included Dakshina (human-annotated Romanized Literary Tamil), social media sets like Tamilmixsentiment, Offenseval, and HopeEDI, and the massive CC-100 web-crawl dataset.
Key Insights: Linguistic Divergence and the "Hidden" Web
Our research quantified the substantial divergence between the two varieties. Using Levenshtein distance, we demonstrated that Spoken Tamil differs significantly from its Literary counterpart, often involving complex phonological changes (such as vowel raising) and heavy use of English loanwords.
One of the most surprising findings was that the CC-100 dataset contains about 23% Spoken Tamil. This explains why models like XLM-R perform well on Spoken Tamil even without explicit training—they have already "seen" it during pretraining. We also noted variation between annotators, highlighting the non-standardized nature of the colloquial variety.
Key Results
- Near-Perfect Identification: Both Naïve Bayes and XLM-R achieved ~99% accuracy in distinguishing between the varieties on the internal IruMozhi test set.
- Generalization: When tested on the external Dakshina dataset, the XLM-R model achieved 81.5% accuracy, significantly outperforming simpler statistical models.
- Dataset Audits: The models confirmed that social media-based datasets (e.g., YouTube comments) are overwhelmingly Spoken Tamil (up to 98%), while Wikipedia-based sets are almost entirely Literary.
Conclusion and Future Work
While IruMozhi is a significant step forward, it currently focuses on the dialect of a specific region (Salem, Tamil Nadu). Future work will aim to increase dialectal diversity and dataset size. The ultimate goal is to move beyond classification toward style transfer—creating models that can automatically translate between formal Literary Tamil and natural Spoken Tamil to make AI more accessible to everyday users.
Summary updated