Introduction
Detecting "indecent" or toxic memes is a notoriously difficult task for artificial intelligence. Unlike standard image classification, memes are deeply multimodal and context-dependent. A picture of a "water cooler" is benign in isolation, but when paired with text mocking a specific demographic, it becomes hateful.
Current models often struggle with two primary challenges: capturing implicit context (sarcasm, irony, and cultural nuances) and identifying explicit relations (real-world connections between entities). While massive Large Vision-Language Models (LVLMs) can handle some of this, they are computationally expensive for real-time moderation. The authors of "Just KIDDIN" introduce KID-VLM (Knowledge-Infused Distilled Vision-Language Model), a compact framework that retains the reasoning capabilities of larger models by leveraging external knowledge.
Methodology
The KID-VLM framework unifies two distinct paradigms—Knowledge Distillation (KD) and Knowledge Infusion (KI)—within a neurosymbolic architecture.
Knowledge Distillation (KD)
The system utilizes a "Teacher" model (LLaVA 1.6-NeXT) to generate detailed captions that capture the implicit reasoning of a meme. A "Student" model, based on CLIP, is trained to align its internal representations with these captions using a consistency loss. This allows the student to mimic the teacher's reasoning without requiring the heavy teacher model during inference.
Knowledge Infusion (KI) and Graph Reasoning
To handle explicit facts, the model queries ConceptNet to extract a sub-graph of entities related to the meme's content. These sub-graphs are processed using a Relational Graph Convolutional Network (R-GCN), enabling "multi-hop" reasoning to connect non-adjacent concepts. Finally, a gated fusion mechanism dynamically weights multimodal features against graph-based knowledge for classification.
Datasets and Experimental Setup
The framework was evaluated on two major benchmarks:
- HatefulMemes Dataset: A standard benchmark of 10,000 memes, tested on both "Seen" and "Unseen" splits to measure generalization.
- HarMeme Dataset: Focused on COVID-19 related harmful content, labeling both the intensity of harm and the target.
Results
KID-VLM established new state-of-the-art (SOTA) records across multiple metrics. On the HatefulMemes (Unseen) split, it achieved a 10.6% improvement in F1 score over previous models like RGCL. On the HarMeme dataset, it reached an AUC of 92.98.
Ablation studies confirmed that removing either the KI or KD components led to significant performance degradation, highlighting the necessity of the hybrid approach. Furthermore, "Hop 2" traversal in the knowledge graph significantly improved performance on unseen data, suggesting that broader context is vital for handling novel toxicity.
Discussion and Conclusion
The paper demonstrates that "fuzzy" neural reasoning from LVLMs and "structured" symbolic knowledge from KGs are complementary. By distilling this knowledge into a ~500M parameter model, KID-VLM provides a computationally efficient solution for real-time moderation.
However, limitations remain. The model's performance is tied to the coverage of ConceptNet; niche subcultures or new pop-culture events may be missed. Additionally, the student model may inherit biases or hallucinations from the teacher LVLM. Future work will focus on improving entity recognition for specific public figures and developing strategies to mitigate inherited biases.
Summary updated