Models that judge whether a meme is hateful must combine a picture with a caption. They usually fail in one of two ways. Either they rely on the words alone, or they merge image and text inside a dense vector space where nobody can tell which pairing of word and picture drove the decision. We propose a middle path. First, translate what the image shows into a small set of readable symbols, such as detected object and attribute names. Then score explicit pairs of the form (text phrase, visual concept) using BERT, a pretrained text encoder that gives each word a vector that depends on its surrounding sentence. The final classifier is a sparse, additive readout over these pairs. It is trained with an attention signal we get for free from the benchmark's own design: counterfactual meme pairs whose label flips when only the image or only the text changes.
This matters for two reasons. The Hateful Memes benchmark was built with such "benign confounders" to defeat single-modality shortcuts. There, a text-only BERT reaches 62.8% accuracy and the best vision-language baseline reaches 69.47%, against 84.70% for humans (The Hateful Memes Challenge). Meanwhile, the standard way to make attention explanations trustworthy is to supervise attention with human-highlighted rationale words. That approach needs expensive token annotations and has never been extended to cross-modal pairings. We claim the confounder structure can stand in for those annotations. We also claim that doing the cross-modal comparison inside one text embedding space, rather than across two encoders, sidesteps the geometric separation between modalities that makes dense fusion hard to audit.
Core Claim
A counterfactual pair shares one modality and differs in the other, yet carries opposite labels. So the reason for the label flip must lie in the (phrase, concept) pairs that differ between its two members. This yields an automatically derived attention target with no human annotation: attention mass should move onto the differing pairs. We claim three things. First, a model whose only cross-modal pathway is a set of explicit, typed (phrase, concept) pair features, supervised this way, will match dense vision-language fusion on Hateful Memes AUROC. Second, it will beat dense fusion specifically on the confounder subset. Third, its explanations will be faithful by construction, because the additive readout means the explanation is the computation. The novel ingredient is the combination of confounder-derived attention targets with a symbolic cross-modal feature space. Neither piece alone is new, and the ablations below separate them.
Relation to Prior Work
Three works motivate the proposal by documenting the gap. The Hateful Memes paper shows that multimodal pretraining barely helps against confounders. Mind the Gap shows that image and text embeddings in CLIP-style dual encoders sit in separate cones, a consequence of network initialization and the contrastive loss. It also shows that the size of this gap measurably changes zero-shot accuracy and demographic harms. This means cross-encoder cosine similarities are a distorted signal. We avoid that distortion by comparing text with names of visual content inside one BERT space, and we test directly whether this helps. Facilitating Multimodal Classification via Dynamically Learning Modality Gap shows that fusion models greedily fit the faster-learning modality. That behavior is exactly what a text-dominant meme classifier displays. They fix it by rescheduling losses, whereas we remove the single-modality pathway to the label.
The method comes from three sources. From supervised attention we take the auxiliary-loss recipe used in HateXplain and in Supervised Rational Attention (SRA). Both penalize the gap between BERT's attention and human rationale masks, and both are text-only. We also reuse HateXplain's rationales to pretrain the phrase-scoring side. We differ from both in where the attention targets come from, namely confounder pairs, and in what is attended over, namely cross-modal pairs. GIST is the closest cross-modal attention-supervision work. It aligns two model-generated visual attention maps in Visual Commonsense Reasoning. Our targets instead come from label-flipping counterfactuals, and our attended units are readable symbols rather than region features. From MAF we take the practice of representing regions by detector class labels and attribute embeddings. MAF also found attribute embeddings of inconsistent value, so we treat attributes as an ablation rather than an assumed win. From symbolic feature engineering, CAAFE and ReaGen, we take the idea of a small, typed grammar of readable features. ReaGen additionally checks proposed features against domain rules. Both works are tabular-only. We carry the grammar idea to cross-modal pairs.
Two works constrain the proposal. MuRGAt found that forcing programmatic grounding raised attribution scores by 9.6 points but cost 7.4 points of answer accuracy. ExPO-HM reports that naive explain-then-detect pipelines underperform direct detection. Both warn that a symbolic bottleneck may impose an accuracy tax. We therefore make "matches dense fusion in AUROC" a falsifiable success criterion rather than an assumption. HateXplain itself warns that plausible attention need not be faithful. Our additive readout is designed to answer that warning, and we test faithfulness by deleting features rather than by assuming it.
Research Program
The pipeline has four stages.
The first stage turns each image into symbols. An off-the-shelf detector and attribute tagger, as in MAF, together with a generic image tagger, produce a set of visual concept names $V$. Recall of this step is measured against captions so that information loss is known.
The second stage extracts candidate phrases $T$ from the meme text. For each pair $(t, v)$, it computes a small vector of typed interaction scores:
- the contextual similarity between $t$'s in-sentence BERT embedding and $v$'s embedding when inserted into the same sentence;
- an "association" score from a masked-language-model probe asking whether $v$ is a likely filler near $t$;
- a "contrast" score that is positive when the text sentiment toward $t$ opposes the typical sentiment toward $v$.
Each pair feature is thus a readable statement, for example "the phrase 'look how many people love you' co-occurs with the visual concept 'empty desert', with a contrast score of 0.8."
The third stage is the classifier. It places an attention distribution $a$ over pairs and outputs $\text{logit}(x)=\sum_{(t,v)} a_{tv}\, w^\top \phi_{tv}$, where $\phi_{tv}$ is the pair's score vector and $w$ is shared. Deliberately, there is no text-only or image-only path. A separate text-only head exists only as a diagnostic.
The fourth stage is training. The objective combines three terms:
$$\mathcal{L} = \mathcal{L}_{\text{task}} + \lambda_1 \sum_{(x,x')} \max\!\big(0,\ m - y_{xx'}(\text{logit}(x)-\text{logit}(x'))\big) + \lambda_2 \sum_{(x,x')} \mathrm{KL}\!\big(\hat a_{x} \,\|\, a_{x}\big)$$
Here $(x,x')$ ranges over confounder pairs, $y_{xx'}\in\{\pm1\}$ is the sign of the label difference, $m$ is a margin, and $\hat a_x$ is the target attention for $x$. That target places uniform mass on pairs that involve the modality which differs between $x$ and $x'$, and near-zero mass elsewhere. The phrase side of the attention is optionally warm-started with SRA-style supervision on HateXplain rationales.
Confounder pairs may not be shipped with explicit links. If not, we recover them by exact matching of the text string (for image-swapped pairs) and by perceptual-hash matching of images (for text-swapped pairs). We report how many pairs are recovered and audit a sample by hand.
Baselines
The naive reference points come from the Hateful Memes paper. Text-only BERT reports 62.8% accuracy and 69.00 AUROC. The strongest dense fusion baseline reported there, Visual BERT pretrained on COCO, reports 69.47% accuracy and 75.44 AUROC. We will rerun Visual BERT COCO on the same splits with the same seed count and a hyperparameter budget equal to ours, and we will quote both the reproduced and the published figures.
The strongest recent method in the file is ExPO-HM. It fine-tunes Qwen2.5-VL-7B with curriculum reinforcement learning and a decision-entropy reward, and it reports F1 gains of up to 15% and 17% over GRPO and DPO baselines. No absolute Hateful Memes AUROC or accuracy for ExPO-HM appears on Lacuna in the file. We will therefore run its released code (JingbiaoMei/ExPO-HM) on our splits and report its F1 and AUROC. We do not expect to beat a 7B generative model on raw accuracy with a BERT-scale model. Against ExPO-HM, success means one of two outcomes: staying within a few AUROC points while exceeding it on confounder-subset accuracy, or exceeding it on deletion-based faithfulness at a small fraction of the compute.
A caption-concatenation baseline feeds a generated image caption plus the meme text into a single BERT. It is the cheapest way to put both modalities in one text space, and it is trained with the same budget.
Four ablations isolate the key ingredients:
- Dense fusion plus confounder supervision. We apply the same pairwise margin loss, and a region-level version of the attention loss, to Visual BERT. This tests whether the confounder signal alone produces the gains.
- Symbolic model without confounder losses. This isolates the effect of the supervision.
- Pair scores from CLIP cross-encoder cosine. Replacing BERT text-text similarity with CLIP image-text cosine tests whether avoiding the modality gap matters.
- Attributes removed. This follows MAF's finding that attributes are unreliable.
"Beating" a baseline means a better mean over five seeds, with non-overlapping bootstrap confidence intervals, on AUROC or on confounder-subset accuracy.
Why not just…?
Why not just caption the image and run BERT on caption plus text? This is the most serious threat, because it also removes the cross-encoder gap. It could plausibly match our AUROC. The difference is structural. The captioned model can still reach its decision from the text alone, and its explanations are dense attention over a concatenated string, with the unfaithfulness HateXplain warns about. Our experiments settle the question in two ways. We train the caption model with the same confounder margin loss and compare. If it matches us on AUROC, confounder accuracy, and deletion faithfulness, then the pair grammar adds nothing and we will report that honestly.
Why not just add a counterfactual consistency loss to a dense vision-language model? This is a direct test of whether symbols matter, and it is our first ablation. The confounder signal may well improve Visual BERT. What a dense model cannot give is an exact decomposition of the logit into readable cross-modal statements. Our claim therefore splits in two. If accuracy is equal, the contribution narrows to faithfulness, which we measure through the deletion tests described below.
Why not just use a large multimodal model that explains itself in natural language, like ExPO-HM? Its free-text explanations are generated outputs, not the computation that produced the label. MuRGAt reports that top models fail to ground roughly 30–35% of their claims. That model family is also orders of magnitude larger. We run ExPO-HM as a baseline and compare explanations on the same deletion test, applied to its cited evidence where possible. We concede in advance that it will likely win on raw F1.
Why not just collect human cross-modal rationale masks? HateXplain's rationale agreement is only Krippendorff's α of 0.46, and Hateful Memes label agreement is a Cohen's κ of 68.4. Human masks are both costly and noisy. Confounder-derived targets are free and follow from label logic. As a check, we will annotate 300 test memes with pair-level rationales and measure how well both our derived targets and the model's attention agree with them.
Proposed Experiments
The main experiment trains on Hateful Memes and reports accuracy and AUROC on the test split. It also reports accuracy restricted to recovered confounder pairs, where a pair counts as correct only if both members are right. This is the metric where shortcut models should collapse.
Faithfulness is measured with two deletion tests:
- Comprehensiveness: the drop in probability when the top-$k$ attended pairs are removed.
- Sufficiency: the probability retained when only those pairs are kept.
Both are computed for all systems, using region or token masking for the dense models. Plausibility is measured as IoU against the 300 hand-annotated memes. For the phrase-scoring side, we also report HateXplain IoU F1 against SRA's reported 0.539. This confirms that warm-starting reproduces text-level alignment before the cross-modal transfer.
For transfer, we evaluate zero-shot and with light fine-tuning on MAMI and PrideMM, the other meme sets used by ExPO-HM. This tests whether the confounder-supervised pair features generalize beyond the benchmark whose construction supplied them.
A modality-gap diagnostic compares the BERT-space pair scores with the CLIP-cosine variant. It also measures, following Mind the Gap, the centroid distance between image and text embeddings in the CLIP variant, to check whether its underperformance, if any, tracks that distance.
The decisive check is a two-way comparison. Is the symbolic model with confounder supervision within one AUROC point of reproduced Visual BERT COCO, and is it better on confounder-pair accuracy and comprehensiveness? And do both advantages vanish when the confounder losses are removed? If the dense model with the same losses closes the confounder gap, the symbolic bridge is only an interpretability device. If neither closes it, the confounder signal is not doing what we claim.
A final information-loss check reruns the symbolic model using captions written by annotators in place of detector output. This bounds how much accuracy is lost to imperfect conversion of images into symbols, as opposed to the bottleneck itself.
References
- The Hateful Memes Challenge: Detecting Hate Speech in Multimodal Memes
- Mind the Gap: Understanding the Modality Gap in Multi-modal Contrastive Representation Learning
- Facilitating Multimodal Classification via Dynamically Learning Modality Gap
- HateXplain: A Benchmark Dataset for Explainable Hate Speech Detection
- Aligning Attention with Human Rationales for Self-Explaining Hate Speech Detection
- Learning to Agree on Vision Attention for Visual Commonsense Reasoning
- MAF: Multimodal Alignment Framework for Weakly-Supervised Phrase Grounding
- Large Language Models for Automated Data Science: Introducing CAAFE for Context-Aware Automated Feature Engineering
- Synergizing Large Language Models and Knowledge-Based Reasoning for Interpretable Feature Engineering
- Multimodal Fact-Level Attribution for Verifiable Reasoning
- ExPO-HM: Learning to Explain-then-Detect for Hateful Meme Detection