Overview of the HASOC Subtrack at FIRE 2023: Identification of Conversational Hate-Speech

Hiren Madhu, Shrey Satapara, Pavan Pandya, Nisarg Shah, Thomas Mandl, Sandip ModhaFIRE (Working Notes) 2023 · OpenReview

TL;DR

This paper introduces the third iteration of the ICHCL task at FIRE 2023, focusing on identifying conversational hate speech in code-mixed languages. The challenge lies in the scarcity of labeled data, prompting the use of unlabeled data for training and a novel dataset designed to improve inter-annotator agreement. The results highlight the effectiveness of deep learning models, with the top submission achieving an F1 score of 0.8079 for binary classification.

What Problem Does This Paper Address?

Online platforms are rife with hate speech, which can incite real-world violence and negatively impact discourse. Identifying this harmful content is crucial for moderation, but it's a complex task. Traditional approaches often focus on isolated posts, missing the nuances of conversational context. Hate speech can be subtle, embedded within replies or dependent on previous messages, making it difficult to detect without considering the entire thread. Furthermore, the prevalence of code-mixed languages (like Hinglish) adds another layer of complexity. The ICHCL (Identification of Conversational Hate Speech in Code-mixed Languages) subtrack aims to address these challenges by providing datasets and tasks specifically designed for conversational hate speech detection in such environments. This year's iteration (ICHCL 2023) specifically tackles the issue of limited labeled data by incorporating unlabeled data into the training process.

Methods

The core of the ICHCL 2023 task revolves around semi-supervised learning. Participants were provided with an unlabeled training dataset and a labeled test dataset. The key requirement was to leverage the unlabeled data to improve model performance on the test set. This approach aims to reduce the reliance on expensive manual annotation.

The paper also details the methodologies of the top-performing teams:

  • FiRC-NLP: This team employed a fine-tuning approach using XLM-ROBERTa-large. For binary classification, they concatenated context with the tweet. For the multi-class task, they first used their binary classifier and then fine-tuned another LLM to distinguish between standalone and contextual hate speech.
  • IRLab@IITBHU: This approach utilized a contrastive loss function to fine-tune a vanilla mBERT model. Features extracted from this model were then fed into a two-layer LSTM to incorporate conversational context.
  • Chetona: This team concatenated different levels of the conversational thread and used IndicBERT for encoding before classification.

A baseline model was also provided, employing pseudo-labeling as a semi-supervised technique. This involved fine-tuning a multilingual BERT model on existing labeled data, predicting labels for the unlabeled data, and then retraining the model on the combined dataset.

Datasets

The paper introduces the ICHCL 2023 dataset, which is an extension of previous iterations (ICHCL 2021 and 2022). The dataset is derived from Twitter (now X) and includes tweets, comments, and responses, forming conversational threads.

  • Key features of the ICHCL 2023 dataset:
  • Focuses on code-mixed Hindi (Hindi written in Latin script).
  • Designed to have higher inter-annotator agreement compared to previous versions.
  • Includes an unlabeled training dataset and a labeled test dataset of approximately 1,000 samples.
  • The data was collected from controversial news stories across various topics to ensure the presence of potentially hateful content.

The participants were also encouraged to use the ICHCL 2021 and 2022 datasets as labeled data for training.

Key Insights and Ideas

  • Context is King (Still): The continued success of models that explicitly incorporate conversational context underscores its critical importance in accurately identifying hate speech. Simply looking at individual posts is insufficient.
  • The Power of Semi-Supervision: The task's emphasis on using unlabeled data highlights a practical and scalable approach to tackling data scarcity in NLP tasks, especially for specialized domains like hate speech detection.
  • Code-Mixing is a Major Hurdle: The focus on code-mixed languages like Hinglish points to the growing need for NLP models that can handle linguistic diversity and the complexities of informal online communication.
  • Inter-Annotator Agreement Matters: The effort to improve inter-annotator agreement in the new dataset suggests a recognition that clearer annotation guidelines and more robust data are foundational for building reliable hate speech detection systems.

Key Results

  • Task 2A (Binary Classification): The top submission by FiRC-NLP achieved an impressive F1 score of 0.8079. Other teams like IRLab@IITBHU and Chetona also demonstrated strong performance.
  • Task 2B (Multi-class Classification): FiRC-NLP again led with a macro F1 score of 0.6541.
  • Baseline Performance: The provided baseline model, while not achieving top scores, offered a starting point for participants.
  • Participant Engagement: Six teams submitted results for Task 2A, and five for Task 2B, indicating a healthy level of participation.
  • Underutilization of Semi-Supervised Methods: Despite the task's design encouraging semi-supervised approaches, no submissions explicitly utilized them, suggesting a potential gap in researcher familiarity or interest in this specific methodology for this task.

Limitations and Future Work

  • Limited Use of Semi-Supervised Methods: A significant limitation noted is that participants did not fully leverage the unlabeled data as intended by the task design. This suggests a need for further research and education on effective semi-supervised techniques for hate speech detection.
  • Transformer Size Constraint: To ensure fairness, participants were restricted to using transformers with fewer than 200M parameters. This limitation might have prevented some teams from exploring larger, potentially more powerful models.
  • Future Directions: The authors suggest data augmentation by large language models as a promising avenue for future evaluations, citing early positive outcomes. This could help further alleviate data scarcity and improve model robustness.

Summary updated

BibTeX
@inproceedings{madhu2023overview,
  author = {Hiren Madhu and Shrey Satapara and Pavan Pandya and Nisarg Shah and Thomas Mandl and Sandip Modha},
  title = {{Overview of the HASOC Subtrack at FIRE 2023: Identification of Conversational Hate-Speech}},
  booktitle = {FIRE (Working Notes)},
  year = {2023},
  url = {https://openreview.net/forum?id=ur3pr01x2r},
}
Figure