Just KIDDIN: Knowledge Infusion and Distillation for Detection of INdecent Memes

Rahul Garg, Trilok Padhi, Hemang Jain, Ugur Kursuncu, Ponnurangam Kumaraguru

Introduction

Detecting "indecent" or toxic memes is a notoriously difficult task for artificial intelligence. Unlike standard image classification, memes are deeply multimodal and context-dependent. A picture of a "water cooler" is benign in isolation, but when paired with text mocking a specific demographic, it becomes hateful.

Current models often struggle with two primary challenges: capturing implicit context (sarcasm, irony, and cultural nuances) and identifying explicit relations (real-world connections between entities). While massive Large Vision-Language Models (LVLMs) can handle some of this, they are computationally expensive for real-time moderation. The authors of "Just KIDDIN" introduce KID-VLM (Knowledge-Infused Distilled Vision-Language Model), a compact framework that retains the reasoning capabilities of larger models by leveraging external knowledge.

Methodology

The KID-VLM framework unifies two distinct paradigms—Knowledge Distillation (KD) and Knowledge Infusion (KI)—within a neurosymbolic architecture.

The KID-VLM architecture, integrating Knowledge Distillation from an LVLM and Knowledge Infusion from a Knowledge Graph.

Knowledge Distillation (KD)

The system utilizes a "Teacher" model (LLaVA 1.6-NeXT) to generate detailed captions that capture the implicit reasoning of a meme. A "Student" model, based on CLIP, is trained to align its internal representations with these captions using a consistency loss. This allows the student to mimic the teacher's reasoning without requiring the heavy teacher model during inference.

Knowledge Infusion (KI) and Graph Reasoning

To handle explicit facts, the model queries ConceptNet to extract a sub-graph of entities related to the meme's content. These sub-graphs are processed using a Relational Graph Convolutional Network (R-GCN), enabling "multi-hop" reasoning to connect non-adjacent concepts. Finally, a gated fusion mechanism dynamically weights multimodal features against graph-based knowledge for classification.

Datasets and Experimental Setup

The framework was evaluated on two major benchmarks:

  1. HatefulMemes Dataset: A standard benchmark of 10,000 memes, tested on both "Seen" and "Unseen" splits to measure generalization.
  2. HarMeme Dataset: Focused on COVID-19 related harmful content, labeling both the intensity of harm and the target.

Results

KID-VLM established new state-of-the-art (SOTA) records across multiple metrics. On the HatefulMemes (Unseen) split, it achieved a 10.6% improvement in F1 score over previous models like RGCL. On the HarMeme dataset, it reached an AUC of 92.98.

Performance comparison on the HatefulMemes dataset, showing KID-VLM achieving SOTA results.

Ablation studies confirmed that removing either the KI or KD components led to significant performance degradation, highlighting the necessity of the hybrid approach. Furthermore, "Hop 2" traversal in the knowledge graph significantly improved performance on unseen data, suggesting that broader context is vital for handling novel toxicity.

Discussion and Conclusion

The paper demonstrates that "fuzzy" neural reasoning from LVLMs and "structured" symbolic knowledge from KGs are complementary. By distilling this knowledge into a ~500M parameter model, KID-VLM provides a computationally efficient solution for real-time moderation.

However, limitations remain. The model's performance is tied to the coverage of ConceptNet; niche subcultures or new pop-culture events may be missed. Additionally, the student model may inherit biases or hallucinations from the teacher LVLM. Future work will focus on improving entity recognition for specific public figures and developing strategies to mitigate inherited biases.

Summary updated

BibTeX
@inproceedings{garg2025just,
  author = {Rahul Garg and Trilok Padhi and Hemang Jain and Ugur Kursuncu and Ponnurangam Kumaraguru},
  title = {{Just KIDDIN: Knowledge Infusion and Distillation for Detection of INdecent Memes}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2025},
}
Figure