TL;DR
This paper introduces WoNBias, a novel, large-scale Bengali dataset designed to detect gender bias and harmful language against women. By combining diverse online and offline data sources, WoNBias aims to address the critical lack of resources for low-resource languages like Bengali, enabling better content moderation and fairer NLP models.
What Problem Does This Paper Address?
The proliferation of online platforms has unfortunately led to an increase in gender-based bias and harmful language directed at women. While significant efforts have been made to detect and mitigate such issues in high-resource languages like English, low-resource languages like Bengali remain underserved. Existing universal tools often fail to capture the nuances of linguistic bias in Bengali due to its flexible grammar and rich idiomatic expressions. This gap leaves women vulnerable to discriminatory narratives online, hindering equitable participation and perpetuating societal inequities. The paper aims to bridge this gap by creating a dedicated dataset for Bengali that reflects its unique sociocultural context.
Methods
The WoNBias dataset was created through a multi-pronged approach:
- Data Collection: Text was gathered from a variety of sources, including public Facebook posts and comments, regional Bengali newspapers (e.g., Prothom Alo), and government articles on gender issues.
- Crowdsourcing and Surveys: To capture real-world experiences, the researchers collaborated with female students and professionals across Bangladesh. They distributed questionnaires to collect personal accounts of gender bias and discrimination.
- Web Scraping: Publicly available online content was meticulously collected, adhering to ethical scraping practices and respecting copyright.
- Annotation: A team of seven university students (four male, three female) from diverse regional backgrounds in Bangladesh annotated the data. This diverse team was crucial for capturing varied interpretations of gender bias influenced by local norms and dialects.
- Quality Assurance: A rigorous quality control protocol was implemented, including inter-annotator agreement checks with a senior annotator to resolve disagreements. Inclusion rules ensured that only self-contained sentences explicitly expressing bias were selected, and derogatory language was flagged. Deduplication was performed to remove identical or near-identical sentences while retaining paraphrases.
Datasets
The primary contribution of this paper is the introduction of the WoNBias dataset.
- WoNBias Dataset: A curated Bengali dataset containing 31,484 annotated text entries. It is categorized into three classes: Negative (10,656 entries), Positive (10,170 entries), and Neutral (10,658 entries). The dataset is publicly available.
- Related Datasets Mentioned:
- BOLD (Dhamala et al., 2021): An English dataset for measuring biases in open-ended text generation.
- BanStereoSet (Kamruzzaman et al., 2024): A Bengali benchmark for measuring stereotypical social biases in LLMs.
- BanglaBERT (Bhattacharjee et al., 2022): A pre-trained language model for Bengali.
- SentNoB (Islam et al., 2021): A dataset for analyzing sentiment in noisy Bengali texts.
- CAMEL dataset: A dataset of culturally grounded Arabic prompts.
Key Insights and Ideas
- The Importance of Culturally Grounded Datasets: The paper strongly emphasizes that universal NLP tools are insufficient for capturing the nuances of gender bias in specific cultural contexts, especially for low-resource languages like Bengali.
- Bridging Online and Offline Bias: By combining data from social media, news, and direct participant surveys, WoNBias offers a more comprehensive view of gender bias as it manifests in both digital and real-world conversations.
- Subtlety of Bengali Bias: The research highlights that Bengali language offers unique avenues for subtle stereotypes and hate speech, which can be easily missed by automated filters.
- Positive Bias is Also Important: The inclusion of a "Positive" category, representing explicit advocacy for women, is crucial for a balanced understanding and for training models that can also recognize and promote positive discourse.
- Inter-Annotator Agreement is Key: The substantial inter-annotator agreement (κ = 0.74) demonstrates the reliability and consistency of the annotation process, despite the inherent challenges in classifying subtle biases.
Key Results
- Dataset Size and Distribution: WoNBias comprises 31,484 annotated entries, with a nearly balanced distribution across Negative (33.84%), Positive (32.32%), and Neutral (33.84%) classes.
- Lexical Diversity: The dataset exhibits significant lexical diversity, with 52,671 unique tokens and over 60 Bengali feminine terms, including derogatory variants. Negative texts showed the highest lexical diversity.
- Model Performance (BanglaBERT Fine-tuning): When fine-tuned on WoNBias, the BanglaBERT model achieved the highest per-class recall for the Neutral class (0.962), followed by Positive (0.881), and Negative (0.824).
- Confusion Matrix Insights: The classifier showed stronger performance in distinguishing Neutral and Positive classes. The Negative class exhibited more confusion, with a notable portion being misclassified as Neutral (10.65%).
- Inter-Annotator Agreement: The Cohen's Kappa score of 0.74 indicates substantial agreement among annotators, validating the dataset's quality.
Limitations and Future Work
- Limitations:
- Binary Gender Focus: The dataset primarily focuses on binary gender bias and does not extensively cover non-binary identities or intersectional discrimination.
- Contextual Bias Detection: The current dataset may struggle with implicit or culturally coded biases that require external context or keyword-based analysis.
- Onomastic Analysis: The absence of onomastic analysis limits the ability to distinguish gendered names or analyze related biases.
- Annotator Bias: The annotators, being university students aged 22-26, may introduce generational, socioeconomic, or urban-centric biases.
- Future Work:
- Cross-Linguistic Expansion: Extend WoNBias to other South Asian languages like Urdu and Hindi.
- Enhanced Dialect Coverage: Incorporate local Bengali dialects (e.g., Sylheti, Chittagonian) to capture bias variations across linguistic subcultures.
- Bias Severity Scale: Develop a scale to classify the intensity of bias, ranging from mild stereotypes to hate speech, to aid targeted content moderation.
- Model Benchmarking: Assess the effectiveness of various language models in addressing gender bias using the WoNBias dataset.
- Addressing Class Imbalance: Explore techniques to handle class imbalance for improved model performance.
- Domain-Specific Fine-tuning: Investigate fine-tuning models on specific domains to enhance bias detection capabilities.
Summary updated