Prompt-enhanced Network for Hateful Meme Classification

Junxi Liu, Yanyan Feng, Jiehai Chen, Yun Xue, Fenghuan LiIJCAI 2024

Abstract

Detecting hateful memes is a critical yet challenging task in multimodal content moderation. Unlike standard image or text classification, memes often rely on the subtle, often ironic, interplay between visual and textual components. This paper presents Pen (Prompt-enhanced Network), a novel framework designed to improve classification by extending prompt learning into the high-dimensional feature space. By utilizing multi-view perception and prompt-aware contrastive learning, Pen aligns target meme features with "demonstrations" of both hateful and non-hateful content. Evaluated on the Facebook Hateful Memes (FHM) and HarM datasets, Pen achieves new state-of-the-art results, demonstrating superior robustness and accuracy in identifying multimodal hate speech.

The Challenge of Multimodal Hate

Hateful memes present a unique "benign-conflation" problem: the text may appear harmless in isolation, and the image may be a standard stock photo, but their combination creates a vitriolic message. Current automated systems often struggle with this nuance. Many existing models attempt to solve this by injecting external knowledge (such as web-searched entity descriptions), but this frequently introduces "noise"—irrelevant data that can confuse the classifier.

Figure 2: Comparison of hateful and non-hateful memes illustrating how text and image context determine the final label.

Furthermore, traditional prompting methods treat the prompt (e.g., "This meme is [mask]") as a simple text input. This fails to exploit the deeper relationships between the meme being analyzed and known examples of hateful versus non-hateful content within the model's internal feature space.

The Pen Framework

The authors introduce the Pen framework to move beyond simple text-based prompting through three sophisticated modules:

1. Regional Information Global Extraction

The input is structured as a sequence containing the "inference instance" (the meme to be classified) and "demonstrations" (randomly selected examples of hateful and non-hateful memes). The system uses LSTMs to extract global features from these specific regions, ensuring the model understands the context of each segment relative to the others.

2. Prompt-enhanced Multi-view Perception (PMP)

Instead of a single pass, Pen employs dual perception networks: a "hateful perception network" and a "non-hateful perception network." It fuses the features of the current meme with the features of the demonstrations using a soft gating mechanism. This allows the model to "perceive" the meme from multiple angles—comparing it against known hate and known non-hate—before making a final judgment.

Figure 5: The complete architecture of the Pen framework, highlighting the Regional Information Global Extraction and Prompt-aware Contrastive Learning modules.

3. Prompt-aware Contrastive Learning (PCL)

To sharpen decision-making, Pen utilizes two types of contrastive loss:

  • Category-oriented: This pulls features of memes with the same label closer together while pushing different labels apart.
  • Prompt-oriented: This specifically aligns the feature of the [mask] token in the inference meme with the feature of the "label" tokens (like [bad] or [good]) used in the demonstrations, anchoring the classification in the feature space.

Experimental Results

The framework was evaluated on two major benchmarks: the Facebook Hateful Memes (FHM) dataset and the HarM dataset (COVID-19 related memes from Twitter).

Pen achieved new state-of-the-art (SOTA) results, particularly in Macro-F1 scores, which are the standard for these imbalanced datasets:

  • On HarM: Pen reached a Macro-F1 of 85.74%, surpassing the previous SOTA (Pro-Cap) by 1.85%.
  • On FHM: Pen reached a Macro-F1 of 75.25%, a 0.66% improvement over Pro-Cap.

Ablation studies confirmed that the Multi-view Perception module is the most critical component; its removal led to the most significant performance drops, proving that comparative "demonstrations" are essential for accurate classification.

Conclusion and Future Directions

Pen demonstrates that prompting in the feature space is a powerful alternative to text-heavy knowledge injection. By focusing on internal feature alignment, the model avoids the redundancy traps of previous methods. While highly effective, the authors note that the choice of "demonstrations" is critical—the model performs best when the demonstrations are topically relevant to the target meme. Future work will explore extending Pen to few-shot tasks and low-resource, text-only classification scenarios.

Summary updated

BibTeX
@inproceedings{liu2024prompt,
  author = {Junxi Liu and Yanyan Feng and Jiehai Chen and Yun Xue and Fenghuan Li},
  title = {{Prompt-enhanced Network for Hateful Meme Classification}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2024},
}
Figure