Introduction
While hateful meme detection has seen significant progress in high-resource languages like English, low-resource languages—specifically Bengali, the seventh most spoken language globally—have been largely overlooked. Researchers have introduced BHM (Bengali Hateful Memes), the first large-scale multimodal dataset for Bengali that identifies both hateful content and its specific targets, such as individuals, organizations, communities, or society.
The Challenge of Multimodal Hate
The challenge in detecting hate in memes is twofold. First, there is multimodal complexity: memes rely on the interplay between images and text. Often, the text is harmless and the image is benign, but together they form a hateful message. Second, target identification is critical for content moderation. Most existing research treats hate detection as a simple binary (hateful vs. not hateful), but knowing who is being targeted is essential for social science research.
Furthermore, Bengali social media is rife with code-mixing (mixing Bengali and English) and code-switching (writing Bengali using English characters), making standard NLP tools less effective.
The DORA Framework
To tackle these challenges, the authors developed DORA (Dual cO-attention fRAmework), a multimodal deep neural network designed to effectively cross-reference visual and textual cues.
Architectural Components
- Feature Extractors: The model uses a CLIP image encoder (Vision Transformer-based) and XGLM (a multilingual generative language model) to handle code-mixed Bengali text.
- Dual Co-Attention Mechanism: Instead of simple concatenation, DORA uses a modified Multi-head Self Attention (MSA) where one modality acts as the "Query" and the other as the "Key/Value."
- VGAR (Vision-Guided Attentive Representation): Visual features weighted by their relevance to the text.
- TGAR (Text-Guided Attentive Representation): Textual features weighted by their relevance to the image.
Datasets and Evaluation
The BHM dataset consists of 7,148 memes collected from Facebook, Instagram, and blogs. Each meme is labeled for hate detection and target identification (Individual, Organization, Community, or Society). The authors also tested DORA on the MUTE (Bengali) and EmoffMeme (Hindi) datasets to prove its generalizability across Indo-Aryan languages.
Key Results and Insights
DORA consistently outperformed nine state-of-the-art baselines, including MMBT, CLIP, and ALBEF:
- Hate Detection: Achieved an F1-score of 0.718, a 4.8% improvement over ALBEF.
- Target Identification: Achieved an F1-score of 0.720, a 5.2% improvement. Notably, the macro average F1 jumped by 13%, showing superior performance in identifying minority classes like Organizations and Communities.
Cross-Lingual Transferability
DORA showed impressive results when trained on Bengali and tested on Hindi. This suggests that the dual co-attention approach captures structural relationships between modalities that transcend specific language boundaries.
Limitations and Future Work
The model still struggles with implicit hate (sarcasm and irony) and lacks "world knowledge." For instance, it may recognize a face but not the specific political controversies associated with it. Future work involves incorporating Named Entity Recognition (NER) and external knowledge bases to better decode local cultural nuances.
Summary updated