TL;DR
This paper introduces HausaHate, the first expert-annotated corpus for detecting hate speech in the Hausa language, a critical step for addressing online abuse in West Africa. They also present baseline models, highlighting the challenges and potential of using advanced language models for this under-resourced language.
What Problem Does This Paper Address?
Online hate speech is a significant problem globally, but research and tools for detecting it are heavily skewed towards English. This leaves many indigenous languages, particularly in regions like West Africa where ethnic conflicts are prevalent, underserved. The authors highlight that existing hate speech detection technologies are not well-developed for African languages, despite the serious impact of online hate speech on social and political stability in countries like Nigeria. This lack of resources hinders efforts to combat online abuse and protect vulnerable communities.
Methods
The core of this work is the creation of the HausaHate corpus. This involved:
- Data Collection: Using the Meta CrowdTangle platform and the Facebook Graph API, the researchers collected approximately 2,000 Facebook comments in Hausa. They focused on keywords related to ethnic groups and terrorism to identify potentially hateful content.
- Expert Annotation: Three native Hausa speakers, who are also NLP experts, manually annotated the collected comments. This annotation was done in two layers:
- Binary Classification: Comments were labeled as either "offensive" or "non-offensive."
- Hate Speech Targets: Offensive comments were further categorized by their target: "race," "gender," or "none."
- Annotation Evaluation: The quality of the annotation was assessed using Cohen's kappa, showing substantial inter-annotator agreement (79% for offensive/non-offensive and 60% for hate speech targets).
- Baseline Models: To demonstrate the utility of the corpus and evaluate current capabilities, the authors fine-tuned several pre-trained language models (LLMs) for Hausa hate speech detection. These included multilingual models like mBERT-cased and AfriBERTa, as well as a Hausa-specific model, XLM-R-base-Hausa. They focused on a binary offensive classification task.
Datasets
The primary dataset introduced in this paper is the HausaHate corpus.
- HausaHate Corpus: Contains 2,000 manually annotated Facebook comments in Hausa.
- Statistics: 678 offensive comments and 1,322 non-offensive comments. For hate speech targets, there are 391 comments targeting "race," 65 targeting "gender," and 222 with "none" as the target.
- Context: Comments were collected from Western African Facebook pages and include context from the original posts and neighboring comments.
Key Insights and Ideas
- The Critical Need for Indigenous Language Resources: The paper strongly emphasizes that effective hate speech detection requires language-specific datasets and models. Relying solely on English-centric tools is insufficient and can perpetuate harm in other linguistic communities.
- Expert Annotation is Key for Nuance: The use of native Hausa speakers with NLP expertise was crucial for accurately annotating the nuances of offensive language and hate speech, especially given the cultural context.
- Multilingual Models vs. Domain-Specific Models: While multilingual models show promise, the results suggest that models specifically pre-trained or fine-tuned on data closer to the target domain (like social media) and language can outperform general multilingual models.
- Racial Hate Dominates: The analysis of the HausaHate corpus revealed that offensive comments are predominantly related to racial hate, underscoring the specific challenges in this region.
- Understanding Political Context is Vital: The authors suggest that a deeper understanding of regional political conflicts is necessary for developing more effective hate speech classifiers for African indigenous languages.
Key Results
- The XLM-R-base-Hausa model achieved the best performance, with an F1-Score of 85.8% for offensive language detection.
- AfriBERTa-base was the second-best performing model.
- The mBERT-cased model performed the worst, likely due to its stronger bias towards English data.
- The corpus statistics show that 67.5% of comments were non-offensive and 32.5% were offensive. Among offensive comments, 57.66% targeted race, 9.58% targeted gender, and 32.74% had no specific target.
Limitations and Future Work
- Limitations:
- The authors acknowledge that the annotation of hate speech targets, while valuable, had a smaller dataset size, which limited their ability to train dedicated target classifiers.
- The reliance on Facebook data means the corpus might not fully represent hate speech on other platforms.
- Future Work:
- The authors suggest expanding the corpus with more data and potentially incorporating other African languages.
- Developing more sophisticated models that can effectively leverage the two-layer annotation (offensive/non-offensive and hate speech targets) is a key direction.
- Further research into understanding the socio-political context of hate speech in West Africa is recommended to improve classifier accuracy.
Summary updated