TL;DR
This paper addresses the critical challenge of detecting abusive content in transliterated Bengali, a common form of online communication that monolingual approaches miss. The authors introduce the first large, publicly available annotated corpus of transliterated Bengali social media comments and evaluate several machine learning and deep learning models, finding SVM to be the most effective baseline.
What Problem Does This Paper Address?
Social media has become a primary platform for user-generated content, but it also harbors a significant amount of abusive, offensive, and hateful language. While considerable research has focused on detecting such content in well-resourced languages like English, low-resource languages like Bengali have lagged behind. A particular challenge in Bengali is the widespread use of "transliterated Bengali," where Bengali words are written using the Latin alphabet. This phenomenon, often combined with English words (code-mixing), makes it difficult for traditional language processing tools, which rely on standard dictionaries and alphabets, to identify abusive content. Existing work on Bengali abusive content detection has largely ignored this transliterated form, leaving a significant gap in understanding and mitigating online harm in this context.
Methods
The researchers employed a multi-stage approach:
- Data Collection: They scraped comments from YouTube, focusing on content related to Bengali dramas and celebrities.
- Data Filtering: A crucial step involved filtering comments to include only those written in transliterated Bengali. This was challenging as language detection tools struggle to differentiate between English and transliterated Bengali. They manually inspected comments, including those with a mix of transliterated Bengali and English if they contained at least two transliterated Bengali words. Comments written solely in the Bengali alphabet were excluded.
- Data Annotation: Two native Bengali speakers annotated the filtered comments into "abusive" and "non-abusive" categories, following guidelines similar to previous work on hate speech and derogatory language. They achieved a substantial inter-annotator agreement (Cohen's kappa score of 0.733).
- Corpus Creation: The final corpus consists of 3000 comments, with 1500 in each category, ensuring a balanced dataset.
- Baseline Evaluation: They evaluated several machine learning (ML) and deep learning (DL) classifiers:
- Traditional ML: Logistic Regression (LR), Support Vector Machine (SVM), and Random Forest (RF). These models used TF-IDF features derived from unigrams and bigrams.
- Deep Learning: A Bidirectional Long Short-Term Memory (BiLSTM) architecture was implemented, utilizing 100-dimensional word embeddings trained on the transliterated corpus.
Datasets
The primary dataset introduced in this paper is a newly created annotated corpus of 3000 transliterated Bengali-English social media comments. This corpus is divided equally into abusive and non-abusive categories. It is made publicly available to aid future research.
Key Insights and Ideas
- Transliterated Bengali is Ubiquitous and Crucial: The paper strongly emphasizes that transliterated Bengali is a prevalent form of communication in Bengali social media, and ignoring it means missing a significant portion of abusive content.
- Monolingual Approaches Fail: The transliterated nature of the text means that words often don't exist in standard Bengali or English dictionaries, rendering traditional monolingual detection methods ineffective.
- The Need for a Dedicated Corpus: The lack of a publicly available, annotated corpus for transliterated Bengali abusive content was a major barrier, which this paper aims to overcome.
- SVM as a Strong Baseline: Among the traditional ML models, SVM demonstrated the highest efficacy, suggesting its robustness for this specific task.
Key Results
- The paper successfully created and annotated the first large corpus (3000 comments) of transliterated Bengali for abusive content detection.
- Among the traditional ML classifiers, SVM achieved the highest F1 score (0.827 ± 0.010), closely followed by LR (0.823 ± 0.006). RF performed slightly lower.
- The deep learning BiLSTM model achieved a lower F1 score (0.790 ± 0.031) compared to SVM and LR. The authors attribute this to the relatively small size of the corpus (3000 comments), suggesting that DL models might require larger datasets to outperform traditional ML methods.
- Word frequency analysis revealed that 76-80% of words in the comments were transliterated Bengali, underscoring the importance of handling this linguistic form.
Limitations and Future Work
- Corpus Size: The authors acknowledge that the corpus size of 3000 comments, while the largest for this specific type of data, is still relatively small, especially for deep learning models.
- Future Work: The authors plan to:
- Expand the size of the transliterated corpus.
- Conduct more rigorous analysis.
- Develop a customized deep learning model to improve performance.
Summary updated