Battling Hateful Content in Indic Languages HASOC '21

Aditya Kadam, Anmol Goel, Jivitesh Jain, Jushaan Singh Kalra, Mallika Subramanian, Manvith Reddy, Prashant Kodali, T. H. Arjun, Manish Shrivastava, Ponnurangam Kumaraguru

Introduction

Detecting hate speech on Online Social Media (OSM) is a persistent "cat-and-mouse" game that becomes significantly more complex in multilingual societies like India. This paper details the approach taken by the PreCog IIIT Hyderabad team for the HASOC 2021 challenge, which focused on identifying hate speech in English, Hindi, Marathi, and code-mixed Hindi-English. The team's strategy, centered on an ensemble of multilingual transformers augmented with specialized classification heads, earned them a 3rd place overall ranking across six subtasks.

Problem Statement

The authors identify three primary hurdles in modern hate speech detection:

  1. Low-Resource Languages: While English is well-supported, regional languages like Marathi lack extensive datasets.
  2. Code-Mixing and Script Switching: The common practice of mixing languages (e.g., Romanized Hindi) often bypasses standard monolingual models.
  3. Conversational Context: Hate speech is frequently contextual; a reply may only appear toxic when viewed alongside its parent post.

Methodology

The proposed pipeline moves beyond standard transformer implementations by fusing deep learning with linguistic feature engineering.

The CNN Head Strategy

A core innovation was the replacement of the standard [CLS] token classification with a custom "CNN Head." The team extracted the last four hidden layers of XLM-RoBERTa (XLM-R), concatenated them, and passed the result through a Convolutional Neural Network (CNN). This architecture, shown in Figure 1, allows the model to capture local n-gram patterns that often signal offensive language more effectively than simple linear layers.

Architecture of the XLM-R model with a CNN classification head.

Handling Class Imbalance and Context

For fine-grained tasks where certain hate categories are rare, the team utilized Focal Loss instead of standard Cross-Entropy. This approach, illustrated in Figure 2 using a Multilingual MiniLM backbone, forces the model to prioritize "hard" or frequently misclassified examples during training.

Model architecture utilizing Focal Loss to address class imbalance in fine-grained classification.

To address conversational context, inputs were formatted as [CLS] <Tweet> [SEP] <Parent Context> [SEP], explicitly modeling the relationship between a post and its predecessor. Additionally, manual features such as profanity percentage, sentiment polarity (via TweetEval), and punctuation distribution were integrated into the final classification.

Datasets and Experimental Setup

The study utilized the HASOC 2021 dataset, covering binary and fine-grained classification for English, Hindi, and Marathi, as well as binary classification for Hindi-English code-mixed conversational threads. Interestingly, the authors found that incorporating historical HASOC data actually degraded performance, likely due to shifts in data distribution over time.

Results and Key Insights

The team's methods proved highly effective, particularly the CNN-augmented XLM-R model.

Language Task Best Method Accuracy Macro F1
English Subtask 1A XLM-R + CNN + Sentiment 81.03% 0.79
Hindi Subtask 1A XLM-R + CNN 80.09% 0.77
Marathi Subtask 1A XLM-R + CNN 88.64% 0.87
Hi-En Code-mix Subtask 2 XLM-R with Norm 69.36% 0.70

Key takeaways from their experiments include:

  • Hashtags as Noise: Removing hashtags improved performance, as they often acted as distractors.
  • Emoji Utility: While emojis didn't impact binary detection, they were crucial for fine-grained classification, providing subtle sentiment cues.
  • CNN Superiority: Using a CNN to process transformer embeddings consistently outperformed simple Multi-Layer Perceptron (MLP) heads by better identifying "trigger" phrases.

Conclusion

The PreCog team's success highlights the importance of architectural customization and specialized loss functions in multilingual hate speech detection. Future work will focus on domain-adaptive pre-training on social media corpora and exploring multi-task learning frameworks to create a more unified, efficient detection system.

Summary updated

BibTeX
@inproceedings{kadam2021battling,
  author = {Aditya Kadam and Anmol Goel and Jivitesh Jain and Jushaan Singh Kalra and Mallika Subramanian and Manvith Reddy and Prashant Kodali and T. H. Arjun and Manish Shrivastava and Ponnurangam Kumaraguru},
  title = {{Battling Hateful Content in Indic Languages HASOC '21}},
  booktitle = {FIRE (Working Notes)},
  year = {2021},
  url = {https://openreview.net/forum?id=Nzd8rew3sdt},
}
Figure