Offensive Language Detection in Code-Mixed Dravidian Languages

21 concepts19 papers1 proposal

Social media users in multilingual regions frequently communicate using a blend of their native languages and English within a single post or comment. This phenomenon, known as code-mixing, presents a unique challenge for automated content moderation. Standard natural language processing (NLP) models are typically trained on monolingual corpora adhering to standard grammatical rules and scripts. In contrast, code-mixed social media comments often feature phonetic typing—where native words are written using the Roman alphabet (transliteration)—alongside English words, local slang, and shifting syntactic structures.

To protect online communities, researchers are developing specialized datasets and modeling strategies tailored to the linguistic fluidity of code-mixed Dravidian languages (such as Tamil, Malayalam, Kannada, and Telugu).

Building Benchmarks from Real-World Toxicity

Because standard language resources fail to capture the nuances of informal online speech, the foundation of this research direction relies on curating gold-standard corpora directly from social media platforms like YouTube.

Initial efforts established baseline datasets to evaluate comment-level toxicity across multiple languages. A benchmark dataset introduced by Bharathi Raja Chakravarthi et al. in Findings of the Shared Task on Offensive Language Identification in Tamil, Malayalam, and Kannada allowed the community to systematically analyze offensive language in Tamil, Malayalam, and Kannada. To explore how related tasks interact, the DravidianCodeMix corpus was leveraged by Bharathi Raja Chakravarthi et al. in Findings of the Shared Task on Multi-task Learning in Dravidian Languages to build multi-task learning frameworks that simultaneously evaluate sentiment and toxicity.

As the field matured, these benchmarks expanded to encompass other under-resourced Dravidian languages. For example, Premjith B et al. organized a shared task detailed in Findings of the Shared Task on Hate and Offensive Language Detection in Telugu Codemixed Text (HOLD-Telugu)@DravidianLangTech 2024, establishing a dedicated pipeline for binary hate speech classification in Telugu-English text.

Overcoming Script and Data Scarcity

A primary hurdle in processing code-mixed text is script variation. A single comment might contain native script, Romanized text, or a chaotic mix of both. If a model only encounters a word in the Tamil script during training, it will fail to recognize the same word typed phonetically in the Roman alphabet on social media.

To bridge this gap, researchers employ targeted preprocessing and data augmentation strategies:

Algorithmic Evolution: From Linear Models to Transformers

How do models learn to detect toxicity amidst this linguistic blending? Early and computationally lightweight approaches relied on traditional machine learning to find statistical patterns in word frequencies. Disne Sivalingam et al. in OffTamil@DravideanLangTech-EASL2021: Offensive Language Identification in Tamil Text applied $\chi^2$ feature selection with linear Support Vector Machines (SVM) and TF-IDF representations to achieve strong baseline performance on Tamil text. Judith Jeyafreeda Andrew similarly demonstrated in JudithJeyafreedaAndrew@DravidianLangTech-EACL2021:Offensive language detection for Dravidian Code-mixed YouTube comments that fine-tuned SVMs could achieve high accuracy across multiple Dravidian languages, often outperforming basic deep learning configurations. Bharathi B et al. established comment-level benchmarks in SSNCSE_NLP@DravidianLangTech-EACL2021: Offensive Language Identification on Multilingual Code Mixing Text to demonstrate the feasibility of these baseline approaches.

Where sequence order matters, recurrent architectures offer an intuitive step forward. Tewodros Achamaleh et al. deployed an RNN-based model in Tewodros@DravidianLangTech 2024: Hate Speech Recognition in Telugu Codemixed Text to capture sequential dependencies in code-mixed sentences.

However, the state of the art has shifted decisively toward pre-trained multilingual language models. Omar Sharif et al. evaluated transformer-based models in NLP-CUET@DravidianLangTech-EACL2021: Offensive Language Detection from Multilingual Code-Mixed Text using Transformers, demonstrating their superior performance over traditional machine learning for multi-class offensive language detection.

Models like multilingual BERT (mBERT) and DistilBERT excel because they share a cross-lingual token space, allowing them to map semantic similarities even when languages switch mid-sentence. For example, Muhammad Zamir et al. in Lidoma@DravidianLangTech 2024: Identifying Hate Speech in Telugu Code-Mixed: A BERT Multilingual and Fida Ullah et al. in Fida @DravidianLangTech 2024: A Novel Approach to Hate Speech Detection Using Distilbert-base-multilingual-cased successfully leveraged multilingual BERT variants to classify hate speech in Telugu-English text.

[Input Text] -> "super video bro, but context waste" (Code-Mixed)
       |
[Multilingual Embedding Space] -> (Aligns "super/video/bro" and "waste" across English/Tamil contexts)
       |
[Transformer Layers] -> (Captures contextual semantics despite script/language switching)
       |
[Classification Head] -> [Offensive / Non-Offensive]

To maximize the utility of these transformers, researchers often combine them with specialized optimization strategies. Because offensive language datasets are frequently plagued by severe class imbalances (where benign text vastly outnumbers hate speech), the authors in OFFLangOne@DravidianLangTech-EACL2021: Transformers with the Class Balanced Loss for Offensive Language Identification in Dravidian Code-Mixed text. integrated a class-balanced loss function into their pipeline to prevent the model from ignoring minority toxic classes. To push performance boundaries further, Debjoy Saha et al. introduced an ensembling strategy optimized via genetic algorithms in Hate-Alert@DravidianLangTech-EACL2021: Ensembling strategies for Transformer-based Offensive language Detection, combining the strengths of multiple distinct transformer architectures.

Granular Localization via Offensive Span Identification

Flagging an entire comment as offensive is a coarse solution that lacks transparency. A more precise paradigm shift involves identifying the exact spans—the specific words or phrases—that contain the toxic content. This granular approach provides an explicit rationale for the model's decision, making content moderation more explainable and allowing platforms to censor specific slurs while leaving the rest of a user's comment intact.

Manikandan Ravikiran et al. pioneered this direction in Findings of the Shared Task on Offensive Span Identification fromCode-Mixed Tamil-English Comments by establishing a shared task for offensive span identification in Tamil-English comments, noting that the most effective architectures combined pre-trained language models with explainability techniques. The authors expanded this token-level sequence labeling paradigm to other languages in Findings of the First Shared Task on Offensive Span Identification from Code-Mixed Kannada-English Comments, creating a verified benchmark dataset for offensive span identification in Kannada-English comments.

By transitioning from sentence-level classification to token-level sequence labeling, these models pinpoint the exact locus of offense within a complex, code-mixed linguistic matrix, paving the way for transparent and explainable content moderation tools.

History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_code_mixed_offensive_language_dataset_c230d373,
  title = {code-mixed offensive language dataset},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/code-mixed-offensive-language-dataset/rdr_c230d373651549019688d724bc06733d},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_code_mixed_offensive_language_dataset_c230d373_v1,
  title = {code-mixed offensive language dataset},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/code-mixed-offensive-language-dataset/rdr_c230d373651549019688d724bc06733d/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake