TL;DR
This paper tackles the challenge of detecting hate speech in low-resource Indo-Aryan languages, which are often overlooked by existing NLP tools. The team developed a robust system by combining a powerful multilingual language model (IndicBERT) with a simpler Naive Bayes classifier and employing a clever data augmentation strategy to improve performance on languages like Assamese, Bengali, and Bodo.
What Problem Does This Paper Address?
The proliferation of hateful content on social media is a significant problem. While tools for detecting hate speech in high-resource languages are becoming more sophisticated, there's a critical gap in research and development for low-resource languages. This paper focuses on Indo-Aryan languages spoken in Eastern India, such as Assamese, Bengali, and Bodo, which lack sufficient data and specialized models for hate speech detection. Existing research often focuses on "offensive" content, which is broader than the more specific and harmful category of "hate speech." This work aims to bridge this gap by developing effective hate speech detection models for these under-resourced languages.
Methods
The core of the team's approach is a two-pronged strategy: data augmentation and model ensembling.
- Data Augmentation: To overcome the scarcity of labeled data, they employed a translation-based augmentation technique. They used the IndicTrans2 model to translate comments from two other languages into the target language. This generated "noisy" training data, which was then added to the original dataset. Emojis were removed, and comments were truncated to 50 tokens to standardize the input.
- Model Architecture and Training: They implemented an ensemble of two models:
- IndicBERT v2: A powerful multilingual language model pre-trained on a large corpus of Indic languages. They used a specific variant (IndicBERT-MLM+Samanantar) fine-tuned for this task. The model's output was passed through a dropout layer and a linear layer for final classification.
- Multinomial Naive Bayes: A simpler, traditional machine learning model trained for each language individually using TF-IDF representations of tokens.
The predictions from the IndicBERT model and the Naive Bayes model were then combined using a weighted average (4:1 ratio) to produce the final hate speech/non-hate speech classification. To further improve generalization, they trained a single ensemble model for all three languages and prepended language and "original" or "translated" tags to the input text.
Datasets
The paper utilized datasets provided for the HASOC 2023 Task 4 competition. The original training data comprised:
- 1281 Bengali samples
- 4036 Assamese samples
- 1679 Bodo samples
The data augmentation process generated additional synthetic training data:
- 5715 Bengali samples
- 2960 Assamese samples
- 5317 Bodo samples
The models were evaluated on test sets for these languages.
Key Insights and Ideas
- Translation Augmentation is Effective for Low-Resource Languages: The strategy of translating data from other languages to augment the training set proved crucial. This technique effectively expanded the dataset and exposed the models to more linguistic variations, leading to improved performance.
- Ensembling Diverse Models Boosts Robustness: Combining a sophisticated transformer model like IndicBERT with a simpler, interpretable model like Naive Bayes yielded better results than either model alone. This suggests that different models capture different aspects of the data, and their combination provides a more comprehensive understanding.
- Contextual Information Matters: Prepending language and "original/translated" tags to the input text helped the model distinguish between different data sources and potentially improve its ability to handle nuances introduced by translation.
Key Results
The team achieved competitive results, placing within the top 10 on the leaderboard for all three languages. Their macro F1 scores were:
- Assamese: 0.7346 (highlighted as the best score for Assamese in the table)
- Bengali: 0.6786
- Bodo: 0.8438
These scores represent significant improvements over existing baselines for these low-resource languages.
Limitations and Future Work
- Limitations: The paper mentions that they experimented with an IndicBERT-SS variant for better lexical sharing but did not observe improvements. The "noisy" training data generated through translation might introduce errors or biases that could affect performance. The paper does not explicitly detail the performance of individual models before ensembling.
- Future Work: The authors suggest that their approach can be extended to other low-resource Indo-Aryan languages. Further research could explore more advanced data augmentation techniques or investigate different ensemble strategies to potentially achieve even higher performance. The goal of promoting a safer and more inclusive online environment for these languages remains a key motivation.
Summary updated