IruMozhi: Automatically classifying diglossia in Tamil

Kabilan Prasanna, Aryaman Arora

Abstract

Tamil, a Dravidian language of South Asia, is a highly diglossic language with two very different registers in everyday use: Literary Tamil (preferred in writing and formal communication) and Spoken Tamil (confined to speech and informal media). Spoken Tamil is under-studied in modern NLP systems compared to Literary Tamil written in the Tamil script, as evidenced by a lack of datasets explicitly targeting the Spoken variety. In this paper, we release IruMozhi, a human-translated dataset of parallel text in Literary and Spoken Tamil. Using IruMozhi, we train classifiers on the task of identifying which Tamil variety a text belongs to. We use these models to gauge the availability of pretraining data in Spoken Tamil, to audit the composition of existing labelled datasets for Tamil, and to encourage future work on the variety.

Introduction

Tamil is a language of two worlds. In formal writing and news broadcasts, one encounters Literary Tamil, a standardized variety that preserves grammatical structures from centuries ago. Conversely, in daily conversation or on social media, Spoken Tamil is used—a variety that differs significantly in phonology, morphology, and lexicon. This phenomenon, known as diglossia, presents a significant challenge for Natural Language Processing (NLP).

Most NLP research for Tamil focuses on the Literary variety because it is the standard for written text (e.g., Wikipedia). However, Spoken Tamil is the primary mode of daily interaction. Because Spoken Tamil lacks a standardized spelling and is often written in the Latin script (Romanized) online, it is frequently ignored or mishandled by models trained primarily on formal text. This creates a performance gap where models proficient in formal Tamil struggle with colloquial inputs.

Methodology

The IruMozhi project aims to bridge this linguistic divide through a multi-step pipeline:

  1. Data Collection & Romanization: Researchers sampled 499 sentences from Tamil Wikipedia (Literary Tamil). Since Spoken Tamil is predominantly found in the Latin script online, the Literary text was automatically transliterated into the Latin alphabet to provide a consistent baseline for comparison.
  2. Human Translation: Two native speakers translated these sentences into Spoken Tamil. Analysis of the resulting dataset showed that translators agreed with each other much more than they agreed with the original Literary text, confirming the distinct nature of the registers.
  3. Augmentation: To account for the informal spelling variations common in Spoken Tamil, rule-based augmentations were applied, expanding the dataset to over 8,000 lines.
  4. Classification: Two types of models were trained: a Naïve Bayes baseline using character and word n-grams, and a finetuned XLM-R (multilingual transformer) model.
Figure 1: Normalized Levenshtein distance distributions showing that Spoken Tamil translations (Ann. 1 ~ Ann. 2) are more similar to each other than to the original Literary text.

Key Insights and Results

The research yielded several critical findings regarding the state of Tamil NLP:

Classification Performance

The finetuned XLM-R model achieved a remarkable 99.4% accuracy on the IruMozhi test set. Furthermore, it demonstrated strong generalization, correctly identifying the Dakshina dataset (out-of-domain Romanized Literary Tamil) as Literary 81.5% of the time.

The "Hidden" Prevalence of Spoken Tamil

A major discovery was that the CC-100 dataset—a standard source for pretraining large models—contains approximately 44% Spoken Tamil. This suggests that modern transformer models may already possess latent knowledge of colloquial Tamil, even without explicit labeling.

Separation of Registers

The impact of finetuning is clearly visible in the model's internal representations. Before finetuning, XLM-R embeddings for Literary and Spoken Tamil were largely overlapping. After training on IruMozhi, the model learned to linearly separate the two varieties into distinct clusters.

Figure 2: UMAP visualization of XLM-R embeddings before and after finetuning. The finetuned model (right) shows a clear separation between Spoken (green) and Literary (blue) clusters.

Auditing Existing Datasets

Auditing common Tamil datasets (Tamilmixsentiment, Offenseval, HopeEDI) revealed they are overwhelmingly composed of Spoken Tamil (85-94%). This highlights the necessity of Spoken Tamil support for critical tasks like sentiment analysis and safety moderation.

Conclusion and Future Work

IruMozhi provides the first human-translated parallel dataset for Tamil diglossia, enabling high-accuracy classification and auditing of existing resources. While the current dataset is limited to the Salem dialect, future work will focus on expanding geographic diversity and developing style transfer models to automatically translate between registers, making formal information more accessible to all Tamil speakers.

Summary updated

Abstract

Tamil, a Dravidian language of South Asia, is a highly diglossic language with two very different registers in everyday use: Literary Tamil (preferred in writing and formal communication) and Spoken Tamil (confined to speech and informal media). Spoken Tamil is under-studied in modern NLP systems compared to Literary Tamil written in the Tamil script, as evidenced by a lack of datasets explicitly targetting the Spoken variety. In this paper, we release IruMozhi, a human-translated dataset of parallel text in Literary and Spoken Tamil. Using IruMozhi, we train classifiers on the task of identifying which Tamil variety a text belongs to. We use these models to gauge the availability of pretraining data in Spoken Tamil, to audit the composition of existing labelled datasets for Tamil, and to encourage future work on the variety.
BibTeX
@inproceedings{prasanna2024irumozhi,
  author = {Kabilan Prasanna and Aryaman Arora},
  title = {{IruMozhi: Automatically classifying diglossia in Tamil}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2024},
  url = {https://openreview.net/forum?id=emlcDnhXim},
}
Figure