Teaching the Image-to-Language Bridge to Read Memes Like Their Community

Many hateful memes contain no hateful words and no hateful picture. The harm shows up only when a harmless caption meets a harmless image, as when a picture of a kitchen is paired with a line about a female astronaut. Vision-language models handle this badly. On the part of MDIT-Bench where text and image are each benign but toxic together, many leading models score near the 20% random-chance baseline. We think one cause sits in a small, neglected component. A vision-language model converts image features into the language model's input space through a small learned layer called the projector. That layer is usually trained only to support literal captions ("a woman standing in a kitchen"). An image passed through it therefore reaches the language model as a literal description, and the community's reading of the image ("women belong in kitchens") has been lost before any reasoning starts.

We propose to retrain only this projector, with no toxicity labels, on image–text posts in which the text is what a community says about the image (captions, replies, reposts) rather than what the image depicts. We then test whether a detector built on the re-aligned model transfers zero-shot to hate benchmarks from platforms, languages and harm types it never saw. The comparison that matters is a controlled one: the same unlabeled data, the same compute and the same trainable-parameter budget, spent on the projector, on the language model, or on the vision encoder. The study asks which component should absorb this kind of specialized pre-training so that the gain carries across platforms instead of staying inside one dataset.

Core Claim

Our hypothesis is as follows. When the projector is adapted with a pragmatic target (community discourse about an image), a detector trained on labels from one source benchmark will transfer zero-shot to unseen meme platforms better than the same model with its original captioning-trained projector. We expect it also to match or beat spending the same budget on low-rank updates to the language model, while losing less general ability. Two things are new. First, domain-specific pre-training is moved from the text encoder, where Singh et al.'s BERT*+ViT put it, to the cross-modal interface. Second, the projector's training text changes from descriptions to discourse. The claim is falsifiable. If the same projector training with descriptive captions does equally well, then the benefit comes from domain images and not from pragmatic alignment. If a caption-then-classify pipeline does equally well, then the projector is not where the problem lies.

Where This Sits

Several lines of work motivate the proposal. MDIT-Bench shows that off-the-shelf multimodal models fail on combinations of benign text and benign images, but it only evaluates models and trains none. ELITE names a "Description Trap" in safety evaluation, where neutral descriptions of harmful scenes confuse scoring. We read this as a symptom of the same literal-description bias, now inside the model rather than the evaluator. ShieldVLM fine-tunes a 7B vision-language model end to end on step-by-step cross-modal reasoning for implicit toxicity. It reaches 89.05% on MMIT statements, but it is tested within its own data distribution and never looks at the projector. We borrow its five correlation modes (semantic drift, contextualization, metaphor, implication, knowledge) as a diagnostic breakdown.

Other work supplies the method. Tsimpoukelli et al. showed that projected image features can serve as a "prefix" of pseudo-words for a frozen language model. Dong et al. showed that updating only the projector, with both the vision encoder and the language model frozen, keeps alignment stable. Video-LLaVA supplies the architecture template: a single shared MLP projector feeding a 7B Vicuna model. It has never been evaluated on safety tasks.

One result constrains the design. Deep Pre-Alignment for VLMs reports that features passed through lightweight projectors stay far from the text space in early language-model layers. A projector-only change may therefore be too shallow. We take this seriously and include a variant that also adapts the first few language-model layers, so that the depth question is measured rather than assumed.

The domain pre-training literature supports specialization. BERTHop and MonoCoder find that targeted in-domain pre-training beats larger general models. ExpGuard shows that general guardrails score near zero F1 when harm is phrased in domain jargon. None of these asks where in a multimodal model the specialization should go.

On the transfer side, Hong & Gauch supply the leave-one-domain-out protocol for hate speech, though for text only. The recommender literature (Language Representations Can be What Recommenders Need) offers the argument that zero-shot transfer works when inputs are mapped into a shared language space instead of platform-specific identifiers. Our analogue is that a community's reading of an image, once expressed in language space, should carry to communities that share the trope but not the platform.

Research Program

Backbone. We build a LLaVA-style model: a frozen CLIP-family vision encoder, a two-layer MLP projector, and a frozen 7B language model, following the Video-LLaVA recipe.

Alignment corpus. We assemble a corpus of image–text posts from public meme and image-reply communities. The text attached to each image is community discourse: overlaid caption, top replies and repost titles. The corpus carries no toxicity labels. It is deduplicated against every evaluation set by perceptual image hash and OCR text overlap. In the strict regime, it also excludes the platforms and languages of the evaluation sets.

Pragmatic alignment objective. The projector is trained with the standard next-token loss, but it predicts the discourse text conditioned on the image. For the same images, we also produce descriptive captions from a captioning model. This yields a matched control: identical images and identical compute, with only the target text changing.

Detector training. After alignment, the projector is frozen again. We train a lightweight detector on labels from a single source benchmark, using identical recipes across all variants. The detector is either a classification head on pooled final hidden states or a short instruction-tuning stage with LoRA (small low-rank weight updates).

Mechanistic check. Following the text-based decomposition of Gandelsman, we map projected image tokens to their nearest language-model vocabulary directions. We then measure whether re-alignment shifts them from depictive words toward stereotype and trope words, especially on benign-confounder pairs.

Baselines

Naive reference. The unchanged model with its original captioning projector, given the same source-label detector training. Every gain is measured against this. Below it sits the text-only floor from the Hateful Memes Challenge: BERT on meme text reached 62.8% accuracy and 69.00 AUROC, and the best multimodal model, Visual BERT COCO, reached 69.47% accuracy and 75.44 AUROC. These are in-distribution numbers, so they bound what source-trained models achieve at home, not zero-shot.

Strongest existing method. ShieldVLM-style end-to-end instruction fine-tuning of the full model on source labels, reimplemented on our backbone. It receives the same labeled source data, the same GPU-hour budget and a tuning grid of equal size. On MMIT it reports 89.05% accuracy on statements, but no reported result was found on Lacuna for ShieldVLM under zero-shot transfer to a different meme benchmark. Our own run of it under leave-one-dataset-out is therefore the reference number. Beating it means higher mean AUROC across held-out targets, with the gap exceeding the across-seed standard deviation, while ShieldVLM keeps its in-domain advantage if it has one.

Domain-pretrained text encoder. A reconstruction of Singh et al.'s BERT*+ViT concatenation. The BERT is continually pre-trained on our alignment corpus's text, so both methods see the same unlabeled data. Its reported in-domain MAMI Task A F1 is 0.874. It represents the claim that domain pre-training belongs on the text side.

Same data, different location. This is the main ablation. The same alignment corpus and objective are spent on LoRA updates to the language model, with rank chosen so that trainable parameters match the projector's. We also spend the same budget on the vision encoder. Two further ablations isolate the proposal's ingredients: the projector re-aligned on descriptive captions of the same images, and projector plus LoRA on the first four language-model layers, which tests the depth concern.

Why not just…?

Why not just fine-tune the whole model on labeled hate data, as ShieldVLM does? This works in-distribution, and that success is exactly what makes it a threat. Its weakness is that source labels encode one platform's tropes and one set of annotators' judgments, and nothing in the file measures how far that carries. Our leave-one-dataset-out runs settle the question directly. If end-to-end fine-tuning transfers as well as projector re-alignment, the proposal's transfer claim fails, and it survives only as a cheaper alternative.

Why not just OCR the text and use a domain-pretrained text classifier? Benign-confounder memes defeat text-only models by construction. That is why text-only BERT reached 62.8% on Hateful Memes, and why about 80 of Singh et al.'s 200 analyzed errors were attributed to insufficient visual grounding. We include this model and report its confounder-pair accuracy separately, where we expect it to sit near chance.

Why not just caption the image with a VLM and classify the caption with a text model? This is the most serious alternative. If the projector's literal bias is the problem, a literal captioner inherits it, and we expect this pipeline to fail on the metaphor and implication modes. That is a prediction, not established evidence. We run the pipeline with the same language model and the same domain-pretrained text classifier. If it matches our method, we will conclude that the bottleneck is not the projector and report that finding.

Why not just continually pre-train the language model on meme text? This is where domain pre-training has usually gone, and Deep Pre-Alignment gives a reason it might be needed. The matched-budget LoRA ablation answers the question with forgetting measured alongside. A tie on transfer with less forgetting would still favor the projector. A clear loss would redirect the program toward deep alignment.

Proposed Experiments

Protocol. Leave-one-dataset-out across four binary meme benchmarks: Hateful Memes, MAMI Task A from Singh et al., ExMute (Bangla and code-mixed, where multimodal models reached only 0.64 accuracy in-domain), and MMIT. Each variant trains its detector on one benchmark and is tested zero-shot on the other three. MDIT-Bench Medium tier serves as a never-trained-on stress test against its 20% chance level.

Metrics. AUROC and macro-F1 per target, plus accuracy on benign-confounder pairs. MMIT results are broken down by ShieldVLM's five correlation modes. Forgetting is measured on general image benchmarks such as MMBench. Everything is reported with three seeds.

Decisive checks. Three results would confirm the claim. First, discourse-aligned projectors beat caption-aligned projectors trained on identical images, with the largest gain in the metaphor and implication modes. Second, the projector matches or beats matched-budget language-model LoRA on held-out targets while forgetting less. Third, the nearest-vocabulary analysis shows confounder images moving toward trope words only after discourse alignment.

The claim fails if either of two results appears. One is caption-then-classify matching our method. The other is gains appearing only when the alignment corpus overlaps the target platform, which would indicate domain adaptation dressed up as zero-shot transfer.