Abstract
Detecting hateful memes represents a significant challenge for artificial intelligence due to the subtle, emergent meaning created by the interplay of text and imagery. This paper introduces RA-HMD (Retrieval-Augmented Hateful Meme Detection), a framework designed to adapt Large Multimodal Models (LMMs) for this task. By utilizing a two-stage fine-tuning process and a retrieval-augmented k-nearest neighbor (KNN) classifier, RA-HMD achieves state-of-the-art performance across six diverse benchmarks. Notably, the framework enhances cross-domain generalization and rationale generation while mitigating catastrophic forgetting of general vision-language capabilities.
Introduction: The Challenge of Meme Detection
Hateful meme detection is often considered a "grand challenge" because hatefulness frequently arises from context-dependent relationships between an image and its caption. While Large Multimodal Models (LMMs) like LLaVA or Qwen2-VL possess significant reasoning potential, they face three primary obstacles in this domain:
- Performance and Overfitting: Standard supervised fine-tuning (SFT) on small meme datasets often leads to overfitting, resulting in poor reasoning and low-quality explanations.
- Generalization Gap: Memes are highly dynamic, evolving with social trends. Models trained on static datasets struggle with "out-of-domain" content.
- Catastrophic Forgetting: Specializing an LMM for meme classification often degrades its performance on general tasks like basic image description or spatial reasoning.
The RA-HMD Framework
The proposed RA-HMD framework addresses these issues through architectural modifications and a novel training strategy.
Architecture and Training
The framework introduces a lightweight MLP Projection Head atop the LMM’s final hidden state to create a dedicated embedding space for classification and retrieval. The training proceeds in two stages:
- Stage 1 (Logistic Regression Augmented SFT): The model is trained using a dual objective: standard text-generation loss and a Logistic Regression Classifier (LRC) loss on the MLP embeddings. Retaining the language modeling loss prevents the model from losing its general conversational abilities.
- Stage 2 (Contrastive Fine-tuning): With the LMM backbone frozen, the MLP and LRC are tuned to optimize the embedding space. The model learns to cluster semantically similar memes with identical labels ("pseudo-gold positives") while distancing those with opposite labels ("hard negatives").
Inference via Retrieval-augmented KNN Classifier (RKC)
During inference, rather than relying solely on a direct model prediction, the system employs a Retrieval-augmented KNN Classifier. It retrieves the $K$ most similar examples from a database of encoded memes and uses a similarity-weighted vote to determine the final classification.
Experimental Evaluation
The framework was evaluated across six diverse benchmarks: HatefulMemes (Meta), HarMeme and Harm-P (COVID-19/Politics), MAMI (Misogyny), MultiOFF (Offensive content), and PrideMM (LGBTQ+ themes).
Results and Key Insights
RA-HMD demonstrates superior performance and robustness compared to both specialized CLIP-based models and general-purpose LMMs.
Key Findings
- State-of-the-Art Performance: Utilizing Qwen2-VL-7B, RA-HMD achieved an AUC of 91.1 on the HatefulMemes dataset, outperforming larger agentic systems.
- Superior Generalization: In cross-dataset evaluations, RA-HMD outperformed standard SFT models by over 20% in AUC in certain scenarios, proving highly effective at handling unseen meme styles.
- Preservation of General Capabilities: Unlike traditional fine-tuning, RA-HMD maintained near-original performance on general benchmarks like MMMU and GQA.
- Improved Rationale Generation: By maintaining the language modeling objective, the framework generates higher-quality explanations for its classifications compared to models that learn only labels.
Conclusion and Future Directions
While RA-HMD significantly advances the state of the art, challenges remain. The system occasionally struggles with extremely subtle visual details and complex cultural nuances. Future work will explore more powerful vision encoders and methods to bridge the gap between AI and human-level understanding of dark humor. Additionally, the inherent subjectivity of hate speech necessitates ongoing human oversight in real-world applications.
Summary updated