Findings of the Shared Task on Offensive Span Identification fromCode-Mixed Tamil-English Comments

Manikandan Ravikiran, Bharathi Raja Chakravarthi, Anand Kumar Madasamy, Sangeetha Sivanesan, Ratnavel Rajalakshmi, Sajeetha Thavareesan, Rahul Ponnusamy, Shankar MahadevanACL 2022

TL;DR

This paper introduces a new dataset for identifying offensive "spans" (specific phrases) within code-mixed Tamil-English social media comments. They organized a shared task to encourage the development of models for this challenging problem, finding that rationale extraction methods, particularly those combining pre-trained language models with explainability techniques, performed best.

What Problem Does This Paper Address?

Online content moderation is crucial for maintaining healthy digital spaces, but it's a complex task, especially with the sheer volume of user-generated content. While much research has focused on classifying entire comments as offensive, identifying the specific parts of a comment that make it offensive (offensive span identification) is more granular and useful for content moderators and automated systems. This is particularly challenging for code-mixed languages, where English and a local language (like Tamil) are interwoven, due to a lack of annotated data. This paper addresses this gap by creating and releasing a dataset of code-mixed Tamil-English comments with annotated offensive spans and by organizing a shared task to spur research in this area.

Methods

The paper describes a shared task with two subtasks:

  • Subtask 1: Supervised Offensive Span Identification: Participants were tasked with identifying offensive spans in test data, given annotated training data. This is framed as a supervised sequence labeling problem.
  • Subtask 2: Semi-supervised Offensive Span Identification: Participants could leverage additional unannotated data alongside the labeled data from Subtask 1.

The winning system from NITK-IT_NLP employed a rationale extraction approach. They trained offensive language classifiers and then used model-agnostic techniques like LIME (Local Interpretable Model-agnostic Explanations) to identify toxic spans as explanations for the classifier's decisions. Another submission from DLRG used a combination of token labeling (BIO tagging) and a BiLSTM-CRF model.

Datasets

The primary dataset used in this work is a newly created dataset of code-mixed Tamil-English social media comments with annotated offensive spans. This dataset was built upon existing work by Ravikiran and Annamalai (2021) and extended with data from Chakravarthi et al. (2022b).

  • Training Data: 4786 comments with 6202 offensive spans.
  • Testing Data: 3742 comments.
  • Subtask 2 Data: 700 additional comments were used for the semi-supervised task.

The paper also mentions the creation of baseline models, including a random baseline and a lexicon-based system, for comparison.

Key Insights and Ideas

  • Code-Mixing is a Major Hurdle: The prevalence of code-switching in Tamil social media makes offensive content moderation significantly more complex than in monolingual English contexts.
  • Span Identification is More Informative: Identifying specific offensive spans provides more actionable information for moderation than simply classifying an entire comment.
  • Rationale Extraction Shines: Approaches that combine offensive language classification with explainability methods (like LIME) to extract rationales for offensiveness proved to be highly effective. This suggests that understanding why something is offensive is key to identifying the offensive parts.
  • Context is King (and Difficult): Offensiveness is highly context-dependent, making it challenging for models to generalize, especially when dealing with sarcasm or subtle expressions.
  • Shorter Sentences are Easier (Sometimes): The DLRG system showed improvement on shorter comments, suggesting that models might struggle with the complexity of longer, more nuanced offensive statements.

Key Results

The shared task saw submissions from two teams.

  • NITK-IT_NLP achieved the best performance with a character-based F1 score of 44.89%.
  • DLRG scored 17.28%, significantly lower than the top team and some baselines.

The paper also presents results for additional runs by NITK-IT_NLP using different pre-trained models (Multilingual-BERT and ELECTRA), which achieved F1 scores of 37.33% and 33.95%, respectively. Notably, the random baseline achieved a respectable F1 score of 39.75%, highlighting the difficulty of the task.

Limitations and Future Work

  • Limited Submissions: Only two teams submitted systems, which is a limitation for drawing broad conclusions about the state-of-the-art. The authors attribute this partly to participants being undergraduate students new to shared tasks and facing time constraints.
  • Dataset Domain: The dataset's origin from YouTube comments, particularly movie trailers, might lead to specific linguistic patterns (e.g., repeated use of certain words) that could influence model performance and generalizability.
  • Contextual Nuances: The paper acknowledges that subtle offensiveness, sarcasm, and context-dependent interpretations remain significant challenges for current models.
  • Future Work: The authors plan to re-run the task with a focus on identifying offensive spans and simultaneously classifying different types of offensiveness. They also aim to explore ways to introduce shared tasks on low-resource Dravidian languages into educational curricula.

Summary updated

BibTeX
@inproceedings{ravikiran2022findings,
  author = {Manikandan Ravikiran and Bharathi Raja Chakravarthi and Anand Kumar Madasamy and Sangeetha Sivanesan and Ratnavel Rajalakshmi and Sajeetha Thavareesan and Rahul Ponnusamy and Shankar Mahadevan},
  title = {{Findings of the Shared Task on Offensive Span Identification fromCode-Mixed Tamil-English Comments}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2022},
}
Figure