OSPC: Detecting Harmful Memes with Large Language Model as a Catalyst

Jingtao Cao, Zheng Zhang, Hongru Wang, Bin Liang, Hao Wang, Kam-Fai WongarXiv 2024 · DOI · arXiv

Abstract

Detecting harmful memes in multicultural environments like Singapore presents unique challenges due to the complex interplay of visual metaphors and multilingual text. This paper introduces the OSPC framework, the winning solution for AI Singapore’s Online Safety Prize Challenge. The system employs a "catalyst" approach, distilling the reasoning capabilities of GPT-4V into a smaller, deployable Qwen-14B model. By integrating advanced image captioning, dual-model OCR, and targeted fine-tuning, OSPC significantly outperforms traditional multimodal benchmarks in identifying social biases across English, Chinese, Malay, and Tamil.

Introduction

Memes represent a significant challenge for AI safety because their harmful nature is often "nuanced and context-dependent," emerging from the interaction between image and text rather than either element in isolation. In multicultural hubs like Singapore, this complexity is compounded by the use of multiple languages and scripts. Existing detection systems, often trained on Western-centric datasets, struggle with low-resource languages like Tamil and the specific visual metaphors prevalent in local contexts.

Methodology

The OSPC framework utilizes a three-step pipeline designed for high accuracy and linguistic flexibility.

The OSPC pipeline architecture, illustrating the flow from multimodal feature extraction to LLM reasoning.
  1. Multimodal Feature Extraction: The system uses the BLIP model for image captioning. For text extraction, a dual-model OCR strategy is employed: PP-OCR handles standard text, while TrOCR serves as a high-confidence fallback for complex fonts or scripts like Tamil.
  2. Linguistic Bridging: To overcome the base LLM's limitations with Tamil, a dedicated Tamil-to-English Transformer translation model ensures the reasoning engine receives high-quality semantic input.
  3. LLM Reasoning (The "Catalyst"): The core classifier is a Qwen-1.5-14B model. To enhance its performance, the authors used GPT-4V to generate detailed reasoning annotations for the Hateful Memes Challenge Dataset. This "distilled" knowledge was then used to fine-tune Qwen via QLoRA.

Data Collection and Augmentation

The model was trained using a combination of established datasets and custom synthetic data:

  • Hateful Memes Challenge Dataset: The primary base for fine-tuning.
  • Synthetic OCR Dataset: To improve multilingual text recognition, the authors generated 52,000 synthetic images using ChatGPT for prompts and Stable Diffusion (SDXL) for image generation, overlaying them with multilingual text in various styles.
  • Regional Corpora: Sourced from Malaysian and Singaporean internet data to train the Tamil translation model.
The synthetic data generation process used to create a robust multilingual OCR dataset.

Results and Key Insights

The OSPC framework achieved first place in the Online Safety Prize Challenge, demonstrating substantial improvements over baseline models:

  • AUROC: 0.7749 (vs. 0.5695 for the FLAVA baseline).
  • Accuracy: 0.7087.

A key insight from this work is the effectiveness of the "Catalyst" strategy. By using GPT-4V as a teacher to explain why a meme is harmful, the researchers successfully transferred sophisticated reasoning logic to a much smaller, more efficient model. Additionally, providing the LLM with explicit definitions of harm categories (e.g., "Socio-Economic Divides") proved more effective than simple binary classification prompts.

Conclusion

While OSPC sets a new benchmark, challenges remain. The reliance on separate captioning models like BLIP can lead to information loss if the model fails to recognize specific cultural or historical figures. Furthermore, the "Tamil bottleneck" persists, as the performance of the student model is inherently limited by the teacher's (GPT-4V) own proficiency in low-resource languages. Future research will focus on "native" multimodal LLMs that can process visual and textual data simultaneously to minimize these gaps.

Summary updated

Abstract

Memes, which rapidly disseminate personal opinions and positions across the internet, also pose significant challenges in propagating social bias and prejudice. This study presents a novel approach to detecting harmful memes, particularly within the multicultural and multilingual context of Singapore. Our methodology integrates image captioning, Optical Character Recognition (OCR), and Large Language Model (LLM) analysis to comprehensively understand and classify harmful memes. Utilizing the BLIP model for image captioning, PP-OCR and TrOCR for text recognition across multiple languages, and the Qwen LLM for nuanced language understanding, our system is capable of identifying harmful content in memes created in English, Chinese, Malay, and Tamil. To enhance the system's performance, we fine-tuned our approach by leveraging additional data labeled using GPT-4V, aiming to distill the understanding capability of GPT-4V for harmful memes to our system. Our framework achieves top-1 at the public leaderboard of the Online Safety Prize Challenge hosted by AI Singapore, with the AUROC as 0.7749 and accuracy as 0.7087, significantly ahead of the other teams. Notably, our approach outperforms previous benchmarks, with FLAVA achieving an AUROC of 0.5695 and VisualBERT an AUROC of 0.5561.
BibTeX
@misc{cao2024ospc,
  author = {Jingtao Cao and Zheng Zhang and Hongru Wang and Bin Liang and Hao Wang and Kam-Fai Wong},
  title = {{OSPC: Detecting Harmful Memes with Large Language Model as a Catalyst}},
  year = {2024},
  eprint = {2406.09779},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2406.09779},
}
Figure