How Much of a Meme's Label Is Actually Multimodal? Mixed-Model Variance Partitioning for Shortcut-Resistant Abuse Detectors

Hateful and misogynistic memes are supposed to be a multimodal problem: a harmless photo and a harmless caption combine into an abusive message. Yet detectors routinely succeed or fail for unimodal reasons. They react to a slur in the text, a familiar face, or a reused image template, and the benchmarks rarely tell us how much of their accuracy comes from such shortcuts. Statistical genetics solved a structurally identical problem decades ago. In a genome-wide association study (GWAS), which scans millions of DNA variants for links to a trait, related individuals share both variants and environments, so naive tests flag variants that are merely markers of ancestry. Linear mixed models fix this. They add a random effect whose covariance is a "kinship matrix" of pairwise genetic similarity, which absorbs the trait variation explained by relatedness, so that only signal beyond relatedness is credited to a variant.

This note proposes to treat meme labels the same way. We build three similarity matrices over a dataset's memes: image-only similarity, text-only similarity, and their elementwise product, which is the analogue of the epistasis kernel in genetics. We then fit a mixed model that splits label variance into an image part, a text part, and an image-by-text interaction part. That decomposition is useful in two ways. As a diagnostic, it estimates how much of each benchmark and each abuse subtype genuinely requires combining the two modalities. As a training signal, the fitted unimodal components are subtracted from the labels before training a small multi-task detector, so the detector's trainable parameters are pushed to model the interaction rather than the shortcut. The detector itself is built under a deployment constraint. One frozen vision-language backbone serves all abuse tasks, and each task adds only a query projection on top of a shared cross-modal key/value context.

Core Claim

The amount of label variance in a meme dataset that is explained by unimodal relatedness can be estimated directly, with the same variance-component machinery GWAS uses for heritability. Removing those unimodal components from the training targets, with out-of-fold estimates, yields detectors that are more accurate on counterfactual "benign confounder" pairs than detectors trained on raw labels. This should hold at equal parameter and compute budgets, and it should beat adversarial or logit-subtraction debiasing whenever shortcuts live in template families of near-duplicate memes rather than in individual features.

The display equation is the whole model. For one task with label vector $y$ over $n$ memes:

$$y = X\beta + u_I + u_T + u_{IT} + \epsilon,\quad u_I \sim \mathcal{N}(0,\sigma_I^2 K_I),\; u_T \sim \mathcal{N}(0,\sigma_T^2 K_T),\; u_{IT} \sim \mathcal{N}(0,\sigma_{IT}^2 K_I \odot K_T),\; \epsilon \sim \mathcal{N}(0,\sigma_e^2 I).$$

Here $X\beta$ holds fixed covariates such as data source and language. $K_I$ and $K_T$ are $n \times n$ cosine-similarity matrices from frozen image-only and text-only encoders. $K_I \odot K_T$ is their elementwise product, and the $\sigma^2$ terms are variance components. The quantity we report is the interaction share $h^2_{IT} = \sigma_{IT}^2 / (\sigma_I^2+\sigma_T^2+\sigma_{IT}^2+\sigma_e^2)$, together with the corresponding unimodal shares $h^2_I$ and $h^2_T$.

Where This Sits in Prior Work

The confounding problem in memes is well documented. The Hateful Memes Challenge built 10,000 memes with hand-made benign confounders: pairs in which the image or the text was minimally changed to flip the label. It showed that text-only BERT already reaches 69.00 AUROC, against 75.44 for the best multimodal model. That paper motivates our work, but its remedy, manual counterfactual construction, does not scale to new languages or platforms. Separating Hate Speech and Offensive Language Classes via Adversarial Debiasing trains an adversary to predict a named confound such as profanity, and reports +5.7% true-positive rate for hate on GermEval18. It is the closest methodological competitor. It differs from our approach in that the confound must be named and must be a per-example property. A kinship-style random effect instead models covariance between examples, which is how template reuse actually shows up. The research file notes the parallel explicitly but finds no paper that applies mixed-model variance partitioning to meme detection.

The statistical method comes from genetics. Fast and flexible linear mixed models for genome-wide genetics (Grid-LMM) fits several random effects, including additive, cage, and epistatic effects in mice, by searching a grid of variance proportions. At each grid point it reuses one matrix decomposition across many traits, which let it scan 20,000 expression traits in 6 minutes. We reuse this trick directly. Every abuse task in a dataset shares the same $K_I$, $K_T$, and $K_I\odot K_T$, so the costly decomposition is computed once and amortized across hatefulness, misogyny, and all four MAMI subtypes, much as Grid-LMM amortized across expression traits. Grid-LMM also constrains us. It is limited to roughly 6 to 8 variance components, and it assumes Gaussian outcomes. Our three components fit comfortably within that limit. Our labels are binary, however, so we adopt the standard liability-scale Gaussian approximation and treat this as a stated limitation, not something we solve. The analogy has a known failure mode in genetics as well: findings transfer poorly across cohorts whose structure differs. We test this directly by checking whether variance estimates hold across languages.

The detector architecture draws on parameter-efficient attention and multi-task work. BAM! Just Like That shares key and value projections across attention experts and keeps queries expert-specific. Even with shared keys and values, it beats attention averaging (perplexity 3.19 for expert KV vs. 3.27 for BTX at 2B scale), and sharing cut latency relative to separate expert caches. That work is unimodal text generation. We transfer the shared-KV, per-task-query design to cross-attention from text over image patches for classification, a setting the file flags as open. Yao et al. show that tuning both query and key projections during adaptation is redundant, which supports training only task queries plus a shared value adapter. On the multi-task side, TADFormer uses task prompts to separate attention maps in dense vision, and LOT Merging merges per-task experts in closed form with a small calibration set. Both are alternatives to our shared-context design. LOT Merging is included as a baseline because it already covers vision-language models (BLIP). The architecture is a supporting choice: the paper's claim rests on the variance partition and the residualized training, and the architecture only needs to be a reasonable parameter-efficient host for them.

Research Program

The first stage is measurement. For the Hateful Memes dataset, the two MAMI subtasks from SemEval-2022 Task 5, the five GOAT-Bench tasks from GOAT-Bench, and the Bangla BANMIME set, we compute $K_I$ and $K_T$ from frozen image-only and text-only CLIP towers. We also compute a template kernel that marks near-duplicate images, by thresholding perceptual similarity, as an alternative to $K_I$. We then fit the three-component model per task on a Grid-LMM-style grid. Bootstrap intervals over memes give the uncertainty on the interaction share and the two unimodal shares. The output is a table of how much each benchmark actually rewards multimodal reasoning. This is publishable on its own, and it is checkable. The Hateful Memes set was constructed to defeat unimodal shortcuts, so it should show a lower unimodal share than MAMI subtask A. If it does not, the kernels are not capturing what we think they capture.

The second stage is training. With five-fold cross-fitting, we compute the best linear unbiased predictions of $u_I+u_T$ for each held-out fold, using models fit on the other folds. We subtract them on the logit scale, so that no example's own label leaks into its correction. The detector is then trained on these residualized targets, mixed with the raw labels at a weight $\lambda$:

$$\mathcal{L} = \lambda\,\ell\big(f(x),\, y - \hat u_I - \hat u_T\big) + (1-\lambda)\,\ell\big(f(x),\, y\big).$$

Here $f(x)$ is the detector's output and $\ell$ is squared error on the residual and cross-entropy on the raw label. The weight $\lambda$ is set per task from the estimated unimodal share rather than tuned on test data. At test time the detector is used alone; the random effects are training-only.

The third stage is the deployment architecture. A frozen CLIP backbone feeds one shared cross-attention block in which text tokens attend over image patches. The key projection is frozen and the value projection carries one low-rank adapter shared by all tasks. Each task contributes only a low-rank query adapter and a linear head. For six tasks this costs well under 1% of backbone parameters, and every task reuses the same image-text context.

Baselines

The naive reference point is the same parameter-efficient detector trained on raw labels with $\lambda=0$. Every claimed gain is measured against it at identical architecture, data, steps, and hyperparameter search budget. The strongest published numbers in the file are the following. Visual BERT COCO reaches 69.47% accuracy and 75.44 AUROC on Hateful Memes, against a human level of 84.70% (The Hateful Memes Challenge). The top MAMI systems reach 0.834 macro F1 on subtask A and 0.731 weighted F1 on subtask B (SemEval-2022 Task 5). GPT-4V reaches 70.29% macro F1 on GOAT-Bench, with open models below 62% (GOAT-Bench). No reported result was found on Lacuna for a CLIP-adapter detector on Hateful Memes, and no exact BANMIME accuracy was found either. We therefore re-run a full CLIP linear-probe-plus-cross-attention model ourselves as the practical strongest open baseline. We count the method as beating it only if confounder-pair accuracy improves beyond bootstrap intervals while overall AUROC and F1 stay within one point.

The most dangerous competitor is logit subtraction against a unimodal bias model. We train image-only and text-only classifiers and subtract their out-of-fold logits, using the same cross-fitting and the same $\lambda$ schedule. This isolates the kinship ingredient. With a linear kernel, the mixed model is close to ridge regression on unimodal features, so any advantage must come from template-family kernels and estimated rather than tuned shrinkage. The second competitor is adversarial debiasing in the style of Separating Hate Speech and Offensive Language Classes via Adversarial Debiasing. Its adversaries predict the label from image-only and from text-only features of the detector's representation, and they get the same tuning budget as our method. Its GermEval18 result (+5.7% hate true-positive rate) sets the expected scale of gain. On the parameter-efficient side, we compare the shared-KV design against two alternatives at matched parameter counts: separate per-task LoRA adapters merged with LOT Merging, and per-task full query/key/value adapters. Two ablations remove the template kernel (using $K_I$ only) and the interaction kernel (estimating unimodal effects without $K_I\odot K_T$ in the model). The second shows whether ignoring the interaction term causes real multimodal signal to be wrongly absorbed and removed.

Why not just…?

Why not just subtract a unimodal model's predictions? This is a real threat. For linear kernels it is nearly the same estimator, and the logit-subtraction baseline above will settle the question. Our method can only win where label correlation runs through clusters of related memes, as with reused templates and near-duplicate captions, which a per-example classifier fits poorly. It also has a second advantage: the variance shares say how much to subtract per task, whereas a bias model needs a tuned weight. If logit subtraction matches us on confounder pairs, we will report the variance partition as a diagnostic contribution only and drop the training claim.

Why not just use adversarial debiasing? It requires naming the confound, and it acts on representations of single examples. The Hateful Memes confounders show that what matters is the relation between a meme and its near-twins. The matched-budget comparison measures this directly. We also expect adversarial training to be less stable with six task heads sharing one small block.

Why not just train on benign confounders or ask a large multimodal model? Benign confounders exist only for Hateful Memes and were costly to make. BANMIME and the code-mixed sets have none, and that is exactly where shortcut learning hurts most. Large models do not escape the problem. GOAT-Bench found GPT-4V at 70.29% macro F1, failing on culturally coded metaphors, with chain-of-thought prompting often degrading open models. We include a zero-shot open multimodal model as a reference, but the deployment target is a cheap shared detector.

Why not just merge per-task experts? LOT Merging reports 82.7% average accuracy on ViT-B/32 vision suites with only 16 to 64 calibration samples. It is therefore a fair parameter-efficient alternative, and we run it. It does nothing about shortcuts, though: it preserves each expert's features, including the unimodal ones. The two methods are complementary, so we also test merging experts trained on residualized targets.

Decisive Checks

The primary metric is confounder-pair accuracy on the Hateful Memes test split: the fraction of hateful/benign-confounder pairs in which both memes are classified correctly. A shortcut-driven detector cannot score well on it. The secondary metrics are AUROC and accuracy on Hateful Memes, macro F1 on MAMI subtask A, weighted F1 on subtask B, and macro F1 on GOAT-Bench.

There are four decisive checks. First, a planted-shortcut test: we stamp a small watermark on 80% of the hateful training memes, then measure test accuracy on memes with the watermark flipped. The template kernel should absorb the watermark, and the raw-label detector should not. Second, a validity check on the diagnostic: the estimated unimodal share should be lower on Hateful Memes than on MAMI subtask A, and per-subtype interaction shares on MAMI subtask B should predict which subtypes show the most category confusion. Third, cross-cultural transfer: we train on English sets and evaluate on BANMIME, where we predict the residualized detector loses less than the raw one. We also check whether the variance shares themselves remain stable across languages, the meme analogue of GWAS portability across ancestries. Fourth, a cost check: inference memory and latency of the shared-KV block against per-task adapters, confirming that the debiasing adds nothing at test time.

The idea fails cleanly if logit subtraction equals our method on confounder pairs and the planted watermark, or if interaction shares are indistinguishable from zero on every benchmark. In the second case the cross-modal kernel would be the wrong measuring instrument, and the note's diagnostic claim would need to be withdrawn.