Many hateful memes are harmful only as a combination. The text is harmless on its own, the image is harmless on its own, and the hateful meaning appears only when the two are put together. The field's main test set for this, the Facebook Hateful Memes benchmark, was built on that premise. It is also public, widely copied, and years old, so a model that scores well on it today may simply remember the answers. English-only training is a separate problem: a detector trained on English memes is routinely deployed on memes in Hindi, Arabic, or code-mixed Nepali, and nobody has measured how well the "meaning from combination" skill carries over. We propose a single evaluation design that answers both questions. We cut existing memes into their image and text parts and pair the parts in new combinations. The new memes are then labeled by people and, where needed, re-rendered with the overlay text in other languages.
The idea is that a recombined meme cannot be memorized as an item, because it never existed, yet every part is familiar. Comparing accuracy on original memes, recombined memes of familiar parts, and memes built from genuinely new parts (post-cutoff imagery and referents) separates three things that current numbers blur together. Those three things are recall of benchmark items, real computation of meaning from the pairing, and ignorance of new cultural referents. Holding the image and meaning structure fixed while changing only the language of the overlay text then turns the same set into a controlled test of zero-shot cross-lingual transfer, which here means applying a model trained only on English labels directly to other languages.
Core Claim
Recombination is a contamination-resistant probe that fits non-literal, subjective multimodal tasks, where existing probes do not apply. The claim is that it yields three separable gaps. The first is a contamination gap: the difference between accuracy on published items and on recombined items built from the same parts. The second is an obsolescence gap: the difference between recombined items and items with post-cutoff parts. The third is a transfer gap: the difference between English and target-language renderings of the same recombined item. The claim also predicts that these gaps rank current detectors differently from their published benchmark scores.
Writing $A_{\text{orig}}$, $A_{\text{recomb}}$ and $A_{\text{new}}$ for paired accuracy on the three item types, the first two gaps are defined as follows:
$$\Delta_{\text{contam}} = A_{\text{orig}} - A_{\text{recomb}}, \qquad \Delta_{\text{obs}} = A_{\text{recomb}} - A_{\text{new}}.$$
The transfer gap $\Delta_{\text{lang}}$ is the same accuracy difference taken between the English and target-language versions of each recombined item. The main unit of scoring is the paired flip. It takes one image with two overlay texts, one making the meme hateful and one making it benign, or symmetrically one text with two images. It counts as correct only if the detector labels both members correctly. A model that relies on one modality alone cannot beat chance on paired flips, so this score isolates the combination skill.
Relation to Prior Work
Four papers motivate this design. The Hateful Memes Challenge introduced "benign confounders", which are minimal image or text swaps that flip the label. It showed that unimodal models fail (text-only BERT 62.8% accuracy, 69.00 AUROC; image-only around 52.7%) against a human accuracy of 84.70%. Its confounders are fixed and public, so we reuse the confounder logic but regenerate the items. Both Text and Images Leaked! showed that multimodal models absorb benchmark text during text-only pretraining and images during alignment. Contamination Detection for VLMs Using Multi-Modal Semantic Perturbations showed that memorized answers are anchored to image pixels, so that a LLaVA-v1.5-7B trained on MMStar rose from about 38% to about 54% on the originals and fell back to about 38% on perturbed images. Together these two predict that simply machine-translating FHM would not remove contamination, since the pixels are unchanged.
Other work constrains the design. The perturbation paper relies on answers grounded in literal visual evidence, and it reports that its diffusion editing fails on typography and culturally specific imagery. Meme meaning lives in exactly those elements. MM-DETECT's option shuffling assumes multiple-choice formats, whereas meme detection is binary. DyVal generates fresh items from graphs, but only for formal tasks with computable answers.
We therefore replace pixel generation with recombination of real parts plus human labeling. From Unveiling Causal Reasoning in LLMs we borrow the idea of a post-cutoff split. That paper built CausalProbe-2024 from 2024 news and showed that models collapse on it. We use the same idea as one arm of a factorial design, not as a whole benchmark.
On the language side, MLQA established the English-to-target transfer gap for text, reporting XLM F1 of 68.0 in Spanish, 54.8 in Arabic, and 48.8 in Hindi. Crosslingual Generalization through Multitask Finetuning showed that English-prompted multitask tuning transfers across 46 languages in mT0 and BLOOMZ. Neither study tests meaning that exists only across modalities. MemeNePAL / NeMeme is the closest multilingual meme work. It trains on native Nepali supervision and shows that English-centric detectors degrade, but it cannot say whether the drop comes from language, culture, or distribution shift. Our parallel renderings fix the culture and distribution and vary only the language. Language Contamination Explains the Cross-lingual Capabilities of English Pretrained Models warns that apparent transfer can itself be leakage. That warning is one reason the transfer test runs on recombined, never-published items.
Research Program
The first step is decomposition. For each FHM dev and test meme, we locate the overlay text with OCR and remove it from the image. The removal borrows the mask-free overlay-text removal approach of CLEAR, a video subtitle-removal method. The output is a clean background image and a text string.
The second step is recombination. We sample new image–text pairs, with priority on pairs that FHM's own confounder structure suggests could flip meaning. Examples are exclusionary phrases paired with images of people versus objects, echoing the refugees-versus-sink example in Pro-Cap. We also add random pairs, which act as a control for how often a new pairing is simply benign. Each new meme gets five annotator judgments. We keep items whose majority label holds up under a Cohen's kappa comparable to the original benchmark's 68.4, and we report agreement per split. Paired flips are assembled only from items that meet this bar.
The third step is the post-cutoff arm. Background images and phrases tied to events after the evaluated models' training cutoffs are collected from news imagery, following the CausalProbe-2024 recipe. They are combined in the same structured way, which yields items that no model could have seen.
The fourth step is language re-rendering. The overlay text of every recombined and post-cutoff item is professionally translated into Spanish, Arabic, Hindi, Chinese, and Nepali. The first four are shared with MLQA so that the multimodal gap can be set against the text-only gap. Nepali allows comparison with NeMeme. The translated text is rendered back onto the identical image in the original meme style. Native speakers mark whether the hateful or benign reading survives translation. Items whose meaning depends on an idiom that does not survive are kept in a separate "non-transferable" bucket, not scored as model failures. That bucket is itself a finding about where language-agnostic representations cannot help.
The fifth step is validating the probe. It is only useful if the contamination gap tracks actual contamination. We fine-tune LLaVA-1.5-7B on the FHM test set at increasing doses, from 10% to 100% of items and for one to three epochs, mirroring the dose-response controls in the two contamination papers above. We require $\Delta_{\text{contam}}$ to rise monotonically with dose while $\Delta_{\text{obs}}$ and the paired-flip score on recombined items stay flat.
Baselines
The naive reference point has three parts. The first is a majority-class predictor. The second is a text-only and an image-only classifier trained on FHM train, namely BERT and a ResNet-style classifier as in the original challenge. These bound what can be achieved without combining modalities, and on paired flips they should sit at chance. The third is the published original-item accuracy of each detector, taken at face value with no correction. This is the number the field currently reports.
The strongest existing detectors are AM3 from Rethinking Multimodal Content Moderation from an Asymmetric Angle and Pro-CapPromptHate from Pro-Cap. AM3 trains its joint representation to differ from both unimodal ones and reports 83.18 AUROC on FHM. Pro-CapPromptHate probes a frozen BLIP-2 with targeted questions and reports 80.87 AUC. They are the strongest competitors because they are explicitly designed to capture meaning that arises from the pairing, so they are the models most likely to hold up on recombination. We also include PromptHate from Prompting for Multimodal Hateful Meme Classification (76.76 AUC with generic captions, as cited in Pro-Cap) and zero-shot LLaVA-1.5. All detectors are trained only on FHM train with their published hyperparameters. Each gets the same small tuning budget of three seeds and a fixed search over learning rate on FHM dev. None sees any recombined, post-cutoff, or translated item before testing. AM3's full-scale training (72 V100s) is out of reach, so we use released checkpoints where available and otherwise a scaled-down retrain, which we label as such.
For cross-lingual transfer, we swap the text backbone of the PromptHate and Pro-Cap pipelines for multilingual encoders, XLM-R and mT0. We train on English FHM only and test on translated renderings. The upper reference is MemeNePAL's in-language supervised F1 of 0.6785 on NeMeme. No reported result was found on Lacuna for any detector on recombined, post-cutoff, or translated FHM items.
The ablation isolating our key ingredient is to take the same translated items but without recombination. These are the original FHM memes with only the text translated. If contamination is anchored to pixels, as the perturbation paper suggests, these will overstate transfer relative to translated recombined items. The difference between the two is the part of apparent cross-lingual ability that is really memorization.
"Beating the baseline" means the following for a measurement proposal. First, the recombination gap must rank detectors differently from original-item accuracy. Second, it must reproduce the dose-response under controlled contamination. Third, it must show a transfer gap that the text-only MLQA gaps do not predict.
Why not just…?
Why not just use MM-DETECT or multimodal semantic perturbations? Option shuffling needs multiple-choice answers, and hateful meme detection is a binary label. Diffusion-edited counterfactual images are the tool the perturbation paper itself says fails on typography and cultural imagery, and those are exactly where meme meaning lives. Recombination uses real, unedited parts, so it does not suffer from generation errors. We will still run a text-only paraphrase perturbation as a comparison probe in the dose-response experiment. If that perturbation tracks contamination as well as recombination does, the simpler probe wins and we will say so.
Why not just scrape fresh multilingual memes? Fresh data removes contamination, and this alternative is a real threat for the narrow question of whether a model is contaminated. But fresh scraped memes change language, culture, topic, and label distribution at once. That is exactly why NeMeme cannot say why English detectors degrade. Our post-cutoff arm contains fresh items, and the factorial design adds what scraping cannot provide: attribution of the drop to memorization, new referents, or language separately.
Why not just use FHM's existing benign confounders, or machine-translate FHM? Both are public and fixed, so they are subject to the same leakage as the rest of the benchmark. Translating the text leaves the memorized image untouched. The ablation in Baselines, which compares original memes with translated text against recombined memes with translated text, directly measures how much this shortcut overstates both combination skill and transfer.
Proposed Experiments and Decisive Checks
The primary experiment reports paired-flip accuracy and AUROC for every detector on four item types: original items, recombined items, post-cutoff items, and translated versions of each. From these we compute the contamination, obsolescence, and transfer gaps.
We have three decisive checks. The first is controlled contamination. LLaVA-1.5-7B fine-tuned on FHM test must show a contamination gap that grows with dose, while its recombined paired-flip score does not improve. Without this, the probe is not measuring contamination. The second is reranking. If AM3 and Pro-CapPromptHate keep their published order and margins on recombined items, then published FHM numbers are trustworthy and the contamination concern is empirically small, which is itself a useful negative result. The third is transfer structure. We regress per-language transfer gaps on the MLQA text-only gaps for Spanish, Arabic, Hindi, and Chinese. A significantly larger multimodal gap, concentrated in paired flips and not in unimodal-hate items, would show that the combination skill transfers worse than plain text understanding. That would be new evidence that language-agnostic text representations do not carry meaning that exists only across modalities.
As a secondary analysis on the post-cutoff arm, we test whether adding on-the-fly web retrieval to a caption-plus-text detector closes the obsolescence gap. This adapts On-The-Fly Information Retrieval Augmentation to meme referents. The analysis also extends the fact-duration question raised by Mitigating Temporal Misalignment by Discarding Outdated Facts, which that paper leaves open for multimodal content.
References
- The Hateful Memes Challenge: Detecting Hate Speech in Multimodal Memes
- Both Text and Images Leaked! A Systematic Analysis of Multimodal LLM Data Contamination
- Contamination Detection for VLMs Using Multi‑Modal Semantic Perturbations
- DyVal: Dynamic Evaluation of Large Language Models for Reasoning Tasks
- Unveiling Causal Reasoning in Large Language Models: Reality or Mirage?
- MLQA: Evaluating Cross-lingual Extractive Question Answering
- Crosslingual Generalization through Multitask Finetuning
- A Multimodal Prompt-based Framework for Analyzing Code-Mixed and Low-Resource Memes
- Language Contamination Explains the Cross-lingual Capabilities of English Pretrained Models
- CLEAR: Context-Aware Learning with End-to-End Mask-Free Inference for Adaptive Video Subtitle Removal
- Pro-Cap: Leveraging a Frozen Vision-Language Model for Hateful Meme Detection
- Rethinking Multimodal Content Moderation from an Asymmetric Angle with Mixed-modality
- Prompting for Multimodal Hateful Meme Classification
- On-The-Fly Information Retrieval Augmentation for Language Models
- Mitigating Temporal Misalignment by Discarding Outdated Facts