Overview of the HASOC Subtrack at FIRE 2021: Conversational Hate Speech Detection in Code-mixed language

Shrey Satapara, Sandip Modha, Thomas Mandl, Hiren Madhu, Prasenjit MajumderFIRE (Working Notes) 2021 · OpenReview

TL;DR

This paper introduces a new challenge, Conversational Hate-Speech in Code-Mixed Languages (ICHCL), focused on detecting hate speech within the context of social media conversations, particularly in languages like Hinglish. The authors developed a dataset and a baseline system, highlighting that considering conversational context significantly improves hate speech detection performance compared to analyzing individual posts in isolation.

What Problem Does This Paper Address?

Social media platforms are rife with hate speech, but detecting it is challenging, especially when users employ code-mixed languages (e.g., a blend of Hindi and English, known as Hinglish). Traditional hate speech detection methods often analyze individual posts without considering the surrounding conversation, which can lead to misclassifications. A post that might seem innocuous on its own could be deeply offensive when viewed within the context of a preceding or subsequent comment. This paper addresses the need to develop systems that can understand and leverage this conversational context to accurately identify hate speech in code-mixed social media dialogues.

Methods

The core of this work is the introduction of the ICHCL subtask and its associated dataset. The authors developed a custom annotation platform to collect and label conversational threads. For evaluation, a baseline system was provided to participants. This baseline system employed a simple concatenation strategy: it combined the source tweet, its comments, and replies into a single text string. This concatenated text was then processed using TF-IDF for feature extraction, followed by a Logistic Regression and a Dense Neural Network classifier. While this baseline is straightforward, it serves as a crucial starting point for understanding the impact of context. The paper also discusses various advanced approaches used by participating teams, many of which leveraged transformer-based models like BERT and its variants, often employing ensemble methods.

Datasets

The paper introduces the ICHCL dataset, specifically designed for Conversational Hate-Speech in Code-Mixed Languages. This dataset was sampled from Twitter, focusing on controversial topics to ensure a rich source of potentially hateful content. It comprises approximately 7000 code-mixed posts (English and Hindi) organized into conversational threads. The dataset was annotated into two classes: (NOT) Non Hate-Offensive and (HOF) Hate and Offensive. The annotation process involved multiple annotators to ensure quality, with an inter-rater agreement of around 74% and a Cohen coefficient of 0.47.

Key Insights and Ideas

  • Context is King (Especially in Code-Mixed Languages): The most significant insight is that conversational context dramatically improves hate speech detection. Posts that might be classified as non-offensive in isolation can be identified as hateful when their surrounding dialogue is considered.
  • Code-Mixing Adds Complexity: The use of code-mixed languages like Hinglish presents a unique challenge for NLP models, requiring them to handle linguistic variations and nuances that are often missed by monolingual systems.
  • Transformer Models Dominate: The performance of participating teams, particularly the top performers, highlights the effectiveness of transformer-based architectures (like BERT variants) in capturing complex linguistic patterns and contextual information.
  • Annotation is Labor-Intensive: Creating high-quality datasets for conversational analysis is a substantial undertaking, requiring careful sampling and rigorous annotation processes.

Key Results

The ICHCL subtask at FIRE 2021 attracted 15 teams and 50 submissions. The primary evaluation metric was Macro F1 score. The best-performing team achieved a Macro F1 score of approximately 0.74. The baseline system, using a simple concatenation and TF-IDF approach, yielded a Macro F1 score of 0.6315. This demonstrates that while sophisticated models are crucial, even a basic contextual approach significantly outperforms non-contextual methods. The paper also notes that many teams outperformed the baseline, with approaches often based on different variants of BERT.

Limitations and Future Work

The authors acknowledge that sampling and annotating social media conversation threads is challenging and requires significant human effort. While the ICHCL dataset is a valuable contribution, the paper suggests future work could involve:

  • Multimodal Features: Incorporating other modalities like images or videos, which are prevalent on social media, could enhance detection capabilities.
  • Summarization: Exploring summarization techniques for conversational dialogues might help in distilling key information for better context understanding.
  • Broader Language Coverage: Expanding the task to include other code-mixed languages beyond Hindi-English.
  • Further Contextualization: Investigating more sophisticated methods for representing and processing conversational context beyond simple concatenation.

Summary updated

BibTeX
@inproceedings{satapara2021overview,
  author = {Shrey Satapara and Sandip Modha and Thomas Mandl and Hiren Madhu and Prasenjit Majumder},
  title = {{Overview of the HASOC Subtrack at FIRE 2021: Conversational Hate Speech Detection in Code-mixed language}},
  booktitle = {FIRE (Working Notes)},
  year = {2021},
  url = {https://openreview.net/forum?id=QMirfSL9v6},
}
Figure