A hateful meme often contains nothing hateful in its picture and nothing hateful in its caption. The harm appears only when the two are read together. Recent work trains vision-language models with reinforcement learning (RL), meaning the model writes its own explanations, a reward scores them, and the model is updated toward higher-scoring outputs. This "explain-then-detect" approach helps on memes, but it is expensive. Every training step must generate several long explanations per meme, one token at a time, and then push gradients through all of them. We suspect most of those tokens are filler such as restated captions, generic hedging, and boilerplate about moderation policy. The few tokens that matter are the ones where the model actually combines the picture with the words.
We propose to find those tokens cheaply and spend the learning signal only on them. For each generated explanation, we score every token twice in parallel. One pass hides the image and keeps the text. The other removes the text overlay and keeps the image. A token counts as "interaction-dependent" only if its predicted probabilities change under both removals. Policy gradients are kept only for these tokens, and whole explanations rich in them are up-weighted. The claim is that this reaches the accuracy of the strongest current meme RL method in fewer GPU-hours. It should also be more robust on the "benign confounder" memes, where a small edit to one modality flips the label. A model that leans on text shortcuts fails exactly there.
Core Claim
Spotlight on Token Perception for Multimodal Reinforcement Learning (VPPO) showed that in visual math reasoning only a minority of tokens depend on the image. Its method measures each token's dependence by the divergence between predictions given the full image and given a randomly blackened image. It then restricts gradients to the top 40% of tokens. This speeds convergence and adds roughly 10% training compute. VPPO was tested only on geometry and math, where the visual signal is literal. In memes that signal is metaphorical, and random blackening also erases the rendered caption, so it mixes the two modalities together.
Our claim is that for emergent-meaning content, the right filter is joint dependence rather than visual dependence. A token that changes only when the image is removed is visual description. A token that changes only when the text is removed is caption paraphrase. Neither carries the meme's emergent meaning, which by definition needs both. Concretely, let $p_t$ be the policy's next-token distribution at position $t$ of a sampled explanation given the full meme. Let $p_t^{-v}$ be the same distribution with the image replaced by a blank and the caption kept in the prompt. Let $p_t^{-x}$ be the distribution with the caption dropped from the prompt and its pixels masked using text-detection boxes. The interaction score is
$$ d_t = \min\left(\mathrm{KL}(p_t \,\|\, p_t^{-v}),\ \mathrm{KL}(p_t \,\|\, p_t^{-x})\right), $$
where KL is the Kullback-Leibler divergence. The minimum is large only when removing either modality changes the token, which is the operational meaning of "needs both." Gradients flow only through the top-$k$ fraction of tokens by $d_t$. Each explanation's group-relative advantage is scaled by its mean $d_t$, following VPPO's trajectory shaping but with this score substituted for VPPO's.
Research Program
We build on ExPO-HM: Learning to Explain-then-Detect for Hateful Meme Detection, using its released code (JingbiaoMei/ExPO-HM). ExPO-HM trains Qwen2.5-VL-7B in three stages. First, a supervised warmup on moderation policy manuals. Second, Group Relative Policy Optimization (GRPO), a critic-free RL method that compares several sampled answers per prompt, run with a curriculum. Third, a conditional decision entropy reward that scores how confidently the post-explanation decision is right or wrong. It reports F1 gains of up to 15 and 17 points over plain GRPO and DPO. Its own stated gap is that it updates all generated tokens uniformly. We keep its warmup, curriculum, and reward unchanged and change only which tokens and trajectories receive gradient. Any difference can then be attributed to the filter.
Cost is central, so the two ablated passes must be cheap. They are teacher-forced: they score an already-generated sequence in a single parallel pass, without new decoding. This is the parallel regime that Is Reinforcement Learning (Not) for Natural Language Processing contrasts with the serial generation loop that dominates RL cost. GRPO with a KL penalty already runs a frozen reference model over every rollout. We batch the two ablated inputs into that same reference forward call. This deliberately departs from VPPO, which uses policy passes. We test whether reference-model scores rank tokens the same way as policy-model scores. The interaction filter should cost roughly twice VPPO's single extra pass, but well under the cost of decoding.
The program has three parts. The first is a measurement study. Hateful Memes is built with known unimodal-hate, multimodal-hate and confounder strata (The Hateful Memes Challenge). We check whether $d_t$ is concentrated in explanations of multimodal hate and confounders, and absent from unimodal-hate cases. If not, the score is not measuring what we claim and the method stops there. The second part is the training comparison under matched budgets. The third part tests whether the gain carries to fine-grained and low-resource settings.
Baselines
The naive reference point is ExPO-HM's own plain GRPO stage with a binary reward and no filtering. ExPO-HM reports that this stage trails the full method by up to 15 F1 points. It sets the floor that any RL variant must clear.
The strongest existing method is full ExPO-HM. The file reports only relative numbers for it: up to +15/17 F1 over GRPO/DPO, +14.4 F1 on attack type, +12.7 F1 on target detection, and an LLM-judge reasoning score of 6.2 versus 5.2 for GRPO. Absolute per-benchmark F1 is not given ("no reported result found on Lacuna for ExPO-HM absolute F1 on HatefulMemes or MAMI"). We will therefore re-run it from the released code on identical splits, backbone, rollout group size and seeds. Its final dev F1 becomes the target. Our method beats it if it reaches that F1 in measurably fewer total GPU-hours, with the ablated passes counted. A second win condition is higher final F1, specifically on the confounder strata, at equal GPU-hours.
The third baseline is VPPO's random-patch-blackening filter dropped into the same ExPO-HM pipeline at $k=0.4$. This ablation isolates our key ingredient, separating interaction-dependence from mere visual dependence. Two further ablations complete the picture. One filters by text-dependence alone. The other keeps a random 40% of tokens, which controls for the regularizing effect of dropping gradients at all.
For external anchors, we use numbers the file reports. On Hateful Memes, VisualBERT COCO reaches 69.47% accuracy and 75.44 AUROC, against 84.70% for humans. On MAMI, MMVAE reaches 0.723 F1 and Poirot reaches 0.759 macro F1 (MMVAE, Poirot). On PrideMM, the frozen-LMM-knowledge classifier reaches 76.31% accuracy (Tzelepi & Mezaris). All RL variants receive the same tuning budget: three values of $k$ and three learning rates each.
Why not just…?
Why not just use VPPO as published? Its blackening masks the rendered caption along with image patches, so its dependency score blends both modalities. It cannot separate a caption paraphrase from an emergent inference. VPPO has also never been run on subjective content. The VPPO-filter ablation settles this directly. If it matches our filter on the confounder strata, our contribution reduces to "VPPO transfers to memes." We would report that result.
Why not just shorten or skip the explanations? This is the real threat. A length budget, or GRPO on direct labels with no explanation, removes most decoding cost and might match F1. ExPO-HM reports that explanations help only after its curriculum fixes them, so the question is open. We include a direct-detection GRPO baseline and a length-capped ExPO-HM at matched GPU-hours. If direct detection matches us on the confounder subsets as well, the efficiency claim fails. What would remain is explanation quality, which we measure with ExPO-HM's judge and human coherence protocol.
Why not just use frozen-model knowledge with a small classifier? Tzelepi & Mezaris train no large model and report 76.31% on PrideMM. However, it generates 20 descriptions per meme at test time and cannot adapt its reasoning to a policy. We compare on PrideMM and Harm-C, reporting accuracy alongside test-time tokens generated per meme.
Why not just fall back to supervised fine-tuning or DPO? The moderation RL study reports roughly 100x data efficiency for GRPO over supervised fine-tuning (Scaling RL for Content Moderation). ExPO-HM reports a 17-point F1 margin over DPO. We include supervised fine-tuning on the manuals as a reference row but do not expect it to be competitive.
Proposed Experiments
The main experiment trains all variants on Hateful Memes and MAMI. We log dev F1 against cumulative GPU-hours, so efficiency is read off curves rather than endpoints. The decisive check is accuracy on paired confounders: a pair counts as correct only if the model labels both the hateful meme and its benign twin correctly. Text-shortcut models score near chance on this metric. Our method must beat both ExPO-HM and the VPPO filter on it.
A second check counts the interaction-dependent tokens kept in correct versus incorrect explanations. If filtering works as claimed, correct explanations of multimodal hate should carry more of them. We also run fine-grained transfer on MAMI's attack categories and PrideMM's targets.
A low-resource stress test uses the Tamil and Malayalam misogyny tasks (DravidianLangTech findings), with shared-task bests of 0.83682 and 0.87631 macro F1. Here code-mixed captions may make caption removal noisy. A failure on this test would bound the method's scope rather than refute it.
Finally, we compare reference-model and policy-model dependency scores. If their token rankings disagree, the cheap batched implementation is invalid. We would then report the true cost of the more expensive policy-pass version.
References
- Spotlight on Token Perception for Multimodal Reinforcement Learning
- ExPO-HM: Learning to Explain-then-Detect for Hateful Meme Detection
- Is Reinforcement Learning (Not) for Natural Language Processing: Benchmarks, Baselines, and Building Blocks for Natural Language Policy Optimization
- The Hateful Memes Challenge: Detecting Hate Speech in Multimodal Memes
- MMVAE at SemEval-2022 Task 5: A Multi-modal Multi-task VAE on Misogynous Meme Detection
- Poirot at SemEval-2022 Task 5: Leveraging Graph Network for Misogynistic Meme Detection
- Improving multimodal hateful meme detection exploiting LMM-generated knowledge
- Scaling Reinforcement Learning for Content Moderation with Large Language Models
- Findings of the Shared Task on Misogyny Meme Detection: DravidianLangTech@NAACL 2025