Abstract
The emergence of social media has led memes to become a powerful mode of communication, blending text, images, and emojis. However, this surge in meme usage has also seen a rise in offensive material. With manual content moderation proving impractical due to the sheer volume of data, there's a pressing need for automated methods to identify harmful memes. Yet, existing research predominantly targets high-resource languages such as English, neglecting low-resource ones like Nepali. To bridge this gap, we introduce the first Nepali meme dataset annotated for hate speech and sentiment. Our contributions are threefold: (1) We create and release NeMeme, a unique dataset featuring Nepali and code-mixed Nepali memes (combining Nepali and English). (2) We evaluate NeMeme using cutting-edge unimodal and multimodal models to establish initial performance benchmarks. (3) We introduce MemeNePAL, a novel multimodal framework employing prompt-assisted learning to effectively categorize Nepali memes. MemeNePAL overcomes the shortcomings of prior state-of-the-art (SOTA) techniques, which were designed for high-resource languages and struggle with Nepali's linguistic differences and cultural subtleties. This work not only promotes inclusivity in content moderation research but also aligns with UN Sustainable Development Goals such as promoting well-being, reducing inequalities, and fostering peace. We adhere to FAIR principles by making the dataset publicly available.
1. Introduction: The Challenge of Low-Resource Multimodal Content
Memes have evolved into a dominant form of online communication, seamlessly blending text, imagery, and cultural context to convey complex ideas. Unfortunately, their viral nature also makes them highly effective vectors for hate speech and offensive content. While automated content moderation has advanced significantly for high-resource languages like English, low-resource languages like Nepali remain largely underserved.
A significant hurdle in moderating Nepali content is the prevalence of "code-mixing." Users frequently blend Nepali, English, and sometimes Hindi within a single meme, creating a linguistic hybrid that breaks traditional Natural Language Processing (NLP) tools. Understanding these memes requires a sophisticated fusion of visual cues with culturally nuanced, low-resource text.
This paper addresses this gap by introducing a specialized dataset and a multimodal architecture designed specifically to decode the subtleties of Nepali and code-mixed memes, aligning with UN Sustainable Development Goals to reduce inequalities and foster peaceful online environments.
2. The NeMeme Dataset
To facilitate research in this domain, the authors introduce NeMeme, the first benchmark dataset of its kind. It consists of 6,234 memes meticulously scraped from diverse social media platforms, including Twitter, Reddit, Instagram, Threads, and Facebook.
Composition and Annotation
The dataset is balanced between 3,180 code-mixed (Nepali-English) memes and 3,054 monolingual Nepali memes. Every entry is dual-labeled for Hate Speech (Hate vs. Non-hate) and Sentiment (Positive, Negative, Neutral). To ensure high quality, the dataset underwent rigorous deduplication and a three-phase annotation process conducted by native speakers. Adhering to FAIR principles, the authors have made this dataset publicly available to the research community.
3. The MemeNePAL Framework
The proposed framework, MemeNePAL (Multimodal meme analysis framework for Nepali memes with Prompt Assisted Learning), utilizes a multi-stage pipeline to extract and fuse visual and textual information:
- Image Captioning Module: Rather than relying solely on raw pixels, the model employs ClipCap to generate descriptive text captions for the meme's image. It extracts CLIP embeddings for the image-text pair and uses a CLIP-S reward metric to ensure the generated caption is contextually relevant.
- Text Processing and Prompting: Embedded text is extracted via Optical Character Recognition (OCR). To help the model grasp linguistic nuances, a multi-step prompting strategy is applied. For half the data, words are randomly masked (Masked Language Modeling, or MLM), while the other half undergoes backtranslation to increase linguistic diversity.
- Multimodal Fusion: The original OCR text, the prompted/backtranslated text, and the generated image captions are fed into a GPT-2 language model. This model acts as the final classifier, integrating all inputs to determine sentiment and detect hate speech.
4. Experimental Results and Key Insights
MemeNePAL was evaluated against various unimodal and multimodal baselines, including popular Large Vision-Language Models (LVLMs) like LLaVA and CogVLM.
Performance Benchmarks
- Hate Speech Detection: MemeNePAL achieved an F1-score of 0.6331 on monolingual Nepali and 0.5497 on code-mixed data, outperforming the previous state-of-the-art (Pro-Cap). On the combined dataset, it reached an F1 of 0.6785.
- Sentiment Analysis: The model hit F1-scores of 0.5301 (monolingual) and 0.4651 (code-mixed), consistently beating all baselines.
- Outperforming LVLMs: Notably, MemeNePAL outperformed recent LVLMs by 3% to 15% in F1-scores. This suggests that while LVLMs are powerful, their training data's bias toward high-resource languages makes them less effective for specialized low-resource tasks.
Key Insights
Ablation studies revealed that Masked Language Modeling (MLM) was the most critical component of the prompt strategy. Forcing the model to predict missing words helps it learn the deep contextual relationships necessary for understanding low-resource text. Furthermore, the results confirm that multimodal fusion is essential; visual context often compensates for the linguistic ambiguity found in code-mixed text.
5. Conclusion and Future Directions
While MemeNePAL sets a new benchmark, challenges remain. The model still struggles with memes requiring deep local political or cultural knowledge. Future work will explore multi-task learning to jointly optimize sentiment and hate speech detection, expand the dataset's cultural coverage, and incorporate interpretability techniques to better understand the model's decision-making process.
Summary updated