DISARM: Detecting the Victims Targeted by Harmful Memes

Shivam Sharma, Md. Shad Akhtar, Preslav Nakov, Tanmoy Chakraborty

Abstract

While automated systems have improved at detecting harmful content, identifying the specific targets of internet memes remains a significant challenge due to the implicit nature of digital humor and the necessity of cultural context. We present DISARM (Detecting vIctimS targeted by hARmful Memes), a multimodal framework designed to identify individuals, organizations, or communities targeted by harmful memes. By integrating entity extraction, external knowledge retrieval, and sophisticated multimodal fusion, DISARM demonstrates superior performance in identifying victims, even when those entities were not encountered during model training.

Introduction

Internet memes have evolved into a primary medium for social commentary, but they are frequently co-opted as vehicles for hate speech, cyberbullying, and disinformation. Current state-of-the-art AI models typically treat meme analysis as a binary classification problem—categorizing a meme as "hateful" or "not hateful." However, this approach fails to answer a critical question for moderation and safety: Who is being attacked?

Identifying victims in memes is exceptionally difficult because the harm is often implicit. Memes rely on satire, sarcasm, and specific political or social contexts that are not immediately apparent from the image or text alone. To address this, we introduce DISARM, a framework that leverages external knowledge to decode the contextual nuances of harmful memes.

The DISARM Framework

DISARM moves beyond simple pattern matching by employing a multi-stage pipeline designed to simulate human-like understanding of social context.

1. Entity Extraction

The system first identifies the actors within the meme using Named Entity Recognition (NER) for text and Person Identification (PID) for visual elements. This ensures the model knows exactly who is being discussed or depicted before evaluating the sentiment directed toward them.

2. Contextualization via External Knowledge

Because memes are often "inside jokes" reflecting current events, internal model weights are rarely sufficient for interpretation. DISARM uses the meme’s text as a search query to retrieve real-world snippets from the web. This "background story" provides the necessary context to understand why a specific depiction might be harmful in a given political or social climate.

3. Multimodal Fusion

To synthesize these diverse data streams, DISARM utilizes a sophisticated fusion architecture:

  • Visual and Contextual Encoding: CLIP is used to encode both the images and the retrieved external context.
  • Textual and Entity Encoding: BERT encodes the meme’s literal text and the identified entities.
  • Low-rank Bi-linear Pooling (LRBP): Rather than simple concatenation, DISARM uses LRBP to allow the image, text, and context modalities to interact deeply. This method captures complex correlations between modalities without the computational explosion associated with standard bilinear pooling.

Experimental Setup and Dataset

To evaluate the framework, we introduced Ext-Harm-P, an extension of the Harm-P dataset. This dataset consists of 4,446 memes related to US politics, re-annotated with specific target names (e.g., "Joe Biden" or "The Democratic Party") rather than coarse categories.

To test real-world generalizability, we utilized a three-tiered evaluation strategy:

  • Test Set A: Entities seen during training.
  • Test Set B: Entities seen during training, but never previously identified as targets of harm.
  • Test Set C: Entities completely unseen during training (Zero-shot scenario).

Results and Key Insights

DISARM significantly outperformed ten strong baselines, including prominent multimodal models such as ViLBERT and MMBT. On the most challenging task—identifying victims among completely unseen entities (Test Set C)—DISARM achieved a macro-F1 score of 0.64, a substantial improvement over the 0.51 achieved by the next-best baseline (MMBT).

Generalization and Interpretability

The success in Test Set C suggests that DISARM is not merely memorizing common targets but is learning the underlying visual and linguistic cues of targeting. Furthermore, the use of attention mechanisms (such as Grad-CAM and BertViz) confirms that the model focuses on relevant visual features and keywords when making its determinations, providing a layer of interpretability to its predictions.

Discussion and Limitations

While DISARM represents a leap forward in victim detection, several challenges remain. The definition of "harm" is inherently subjective, and the line between fair political satire and harmful attack can be thin. Additionally, the model’s performance is contingent on the quality of external search results; irrelevant or biased search data can lead to reasoning errors. Finally, while the current study focuses on US politics, further research is required to adapt the framework to different cultural contexts and diverse forms of harm, such as misogyny or localized social conflicts.

Conclusion

By combining multimodal analysis with real-time external knowledge, DISARM provides a robust framework for identifying the targets of harmful memes. This shift from binary classification to specific victim identification is a vital step toward more nuanced and effective content moderation in the digital age.

Summary updated

BibTeX
@inproceedings{sharma2022disarm,
  author = {Shivam Sharma and Md. Shad Akhtar and Preslav Nakov and Tanmoy Chakraborty},
  title = {{DISARM: Detecting the Victims Targeted by Harmful Memes}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2022},
  doi = {10.18653/v1/2022.findings-naacl.118},
}
Figure