Typed Discrepancy Groups: Splitting Cross-Modal Attention by How a Post Lies

A lot of multimodal misinformation involves no doctored pixels and no false sentence on its own. A real photo is paired with a caption naming the wrong person, the wrong event, or the wrong year. Detectors for this kind of fake usually compare text and image with attention, a mechanism that scores how strongly each word relates to each image region. Standard attention is built to find agreement. A sensational but loosely related photo therefore looks "consistent" to it, a failure the literature calls pseudo-consistency. Current fixes either invert the whole attention map or build separate subnetworks. Neither matches the structure of the problem: fakes differ in what is mismatched (an entity, an event, a time), and a single detector blurs these together.

We propose to split the channels of a cross-modal attention layer into small groups, as grouped-attention work does for efficiency. Each group is assigned one kind of mismatch and trained, using fine-grained attribution labels, to measure only that mismatch. A few groups stay ordinary "agreement" groups. The discrepancy groups share one small "noise" attention map that they subtract out, following the asymmetric head sharing of grouped differential attention. In two sentences: the attention layer is partitioned into typed subspaces, each asked a different question about how the caption and image disagree. We expect this to detect fakes about as well as dense inconsistency models at lower cost, to name the type of fake without a separate module, and to interfere less with itself when trained on mixed sources, where current detectors are known to forget.

Core Claim

In misinformation detection, channel grouping inside attention is useful mainly as a way to keep evidence about different kinds of deception in separate, supervised subspaces, and only secondarily as a compression trick. The claim is falsifiable. If a dense attention layer with the same parameter count and the same attribution supervision attached as output heads matches the grouped model, then grouping adds nothing, and we will say so.

Relation to Prior Work

Grouped attention supplies the mechanism. LW-Transformer splits attention and feed-forward projections into $k$ independent groups for vision-and-language tasks. It cuts projection parameters by 45% while slightly improving GQA and RefCOCO, which suggests grouping can regularize rather than just compress. Its groups are uniform and untyped, and it was only tested on cooperative image–text tasks where both modalities are assumed authentic. Grouped Differential Attention makes differential attention (a signal attention map minus a noise map) asymmetric by sharing a few noise heads across many signal groups. Ratios of 3:1 and 4:1 help, while an extreme 11:1 ratio hurts by 2.01%. This constrains our design: the shared noise map cannot be starved, so the ratio becomes a swept hyperparameter rather than a fixed choice. EfficientViT's cascaded group attention feeds each head a different channel slice to reduce redundancy between heads. It also shows that saved FLOPs do not always become saved wall-clock time. GRFormer reports the same gap from tensor splitting overhead. We therefore report measured latency, not only MACs.

Misinformation work motivates the typed split. MIAN subtracts attention affinities from a constant to amplify conflicting cross-modal patterns and reaches 93.6% accuracy on Weibo17. Its inversion is a single global map over full-dimensional projections, not a set of specialized subspaces. The dual-inconsistency network separates modal-shared from modal-unique features with an orthogonality loss and adds knowledge-graph contradiction. It splits at the whole-vector level and fuses late; we split inside the attention projections themselves. MGCA on the AMG benchmark shows that fakes fall into distinct types: image fabrication, non-evidential image, entity, event, and time inconsistency. It handles each view with a separate large module (Vicuna, LLaVA, reverse image search). We ask whether a single attention layer with typed channel groups can recover most of that attribution ability, excluding the parts that truly require outside evidence. FMS already attends to both consistent and inconsistent interactions under multiple supervisions on DGM$^4$. It is the closest in spirit, but its supervision acts on outputs and grounding rather than on partitioned attention subspaces. Finally, MMFakeBench reports that specialized detectors forget earlier forensic cues when fine-tuned on mixed sources. That observation motivates our interference test.

No paper in the file puts grouped attention into misinformation detection. The obvious combination would be to group the four dense co-attentions of Factify systems purely for speed. We deliberately do not make that our contribution, because it would change cost and not what the model can detect.

Mechanism

Text tokens (from a BERT-style encoder) and image patches (from a ViT) enter a cross-attention block whose channel dimension $C$ is split into $G$ groups of width $C/G$. $G_a$ of them are agreement groups using ordinary softmax attention. The remaining $G_d$ are discrepancy groups, one per mismatch type $t$ (entity, event, time, fabrication, non-evidential). The discrepancy groups share one noise projection. For discrepancy group $g$ the attention map is

$$A_g = \mathrm{softmax}\!\left(\frac{Q_g K_g^\top}{\sqrt{d}}\right) - \lambda_g\,\mathrm{softmax}\!\left(\frac{Q_n K_n^\top}{\sqrt{d}}\right),$$

where $Q_g, K_g$ are the group's query and key projections, $Q_n, K_n$ are the shared noise projections, $d = C/G$, and $\lambda_g$ is a learned scalar. Each discrepancy group pools its output into a score $s_g$. Training minimizes

$$\mathcal{L} = \mathrm{BCE}(\hat y, y) + \beta \sum_{g} \mathrm{BCE}(s_g, y_{t(g)}),$$

where $y$ is the real/fake label, $\hat y$ is the classifier output from all groups, $y_{t(g)}$ indicates whether the post has mismatch type $t(g)$, and $\beta$ weights attribution supervision. On datasets without type labels, only the first term is used, and the groups keep the specialization they acquired on AMG.

Time inconsistency cannot be seen from the image–caption pair alone. MGCA obtains it from reverse image search. We will feed retrieved publication metadata as extra text tokens when it is available, and otherwise report the time category separately rather than hide the gap.

Research Program

We first train on AMG, the only file benchmark with per-type attribution labels. We then fine-tune detection-only on Weibo17, Weibo21, GossipCop, PolitiFact, Fakeddit, and DGM$^4$, and evaluate zero-shot and fine-tuned on MMFakeBench's mixed-source set. Three analyses follow. The first measures specialization directly: we ablate one discrepancy group at a time and check that accuracy falls mainly on its assigned type. The second sweeps the agreement-to-discrepancy ratio and the noise-sharing ratio, expecting the non-monotone behavior that GDA saw at extremes. The third is an interference test. We train sequentially on DGM$^4$ (face swaps and text attribute edits) and then AMG, and measure how much DGM$^4$ performance survives. We also track how much each group's weights move, predicting that groups unrelated to the new data move least.

Baselines

The naive reference is frozen text and image embeddings concatenated with their cosine similarity and passed to an MLP. FACTIFY reports that this family reaches 53.09% F1 with a Random Forest. We rerun it with our encoders so that any gain is attributable to attention and not to better features.

The strongest detection competitor on social-media benchmarks is MIAN, which reports 93.6% accuracy on Weibo17, 93.8% on Weibo21, 92.3% on GossipCop, 95.3% on PolitiFact, and 0.972 on Fakeddit manipulated content. It is the right opponent because it targets the same pseudo-consistency failure. We reimplement it on the same BERT and ViT backbones, splits, and hyperparameter search budget. Beating it means matching its accuracy within one standard deviation over five seeds with fewer attention parameters and lower measured latency, or exceeding it outright.

On AMG, the reference is MGCA at 83.23% detection accuracy and 73.85% attribution accuracy. MGCA uses large external models, so matching it is not required. Our target is to beat BMR, which MGCA outperforms by 2.5%, on detection, and to reach attribution accuracy within a few points of MGCA on the four types that need no retrieval, at a small fraction of its inference cost. On DGM$^4$ the reference is FMS, at 96.46% AUC and 93.43% multi-label mAP. On MMFakeBench, the reference is GPT-4V with MMD-Agent at 61.6% F1. Beating it with a model small enough for streaming would be a strong result, though we do not assume it.

The decisive ablation is a dense-attention model with the same backbone and parameter count, with the same attribution labels supervising five output heads on top of the fused representation. A second ablation keeps grouping but assigns types to groups at random, or drops the type loss entirely. This separates the effect of structure from the effect of supervision. A third removes the shared noise map, leaving independent discrepancy groups.

Why not just…?

Why not just add attribution heads to a dense detector? This is the real threat. If supervision alone drives the benefit, where it attaches should not matter. The matched dense multi-task ablation settles it. Our prediction is that the two models tie on in-distribution accuracy and diverge on the single-group removal test and on the sequential DGM$^4$→AMG interference test, where dense models have no subspace to protect. If they also tie there, the core claim fails.

Why not just use MIAN's inverse attention, which already reaches over 92% on several benchmarks? Those benchmarks are close to saturated, so accuracy on them cannot tell the two methods apart. MIAN's single inverted map also gives no account of which mismatch it found. The comparison that matters is AMG attribution and MMFakeBench mixed-source F1. MIAN reports neither, and we run it on both.

Why not just prompt a large vision-language model with retrieval tools? MMFakeBench shows GPT-4V at about 51% F1 with plain prompting and 61.6% with the MMD-Agent loop, while LLaVA-1.6-34B goes from 25.7% to 49.9%. That approach is costly per post and still weak. It remains the right tool for time inconsistency, which needs outside evidence. We treat it as complementary: our model could act as a cheap first-pass filter that also routes posts to the agent by predicted type.

Why not just compress an existing detector with pruning or distillation? Compression preserves whatever the dense model learned, including its bias toward agreement. It does not create typed subspaces. We include a pruned MIAN at matched parameter count as a latency reference. If it matches our accuracy and latency, the efficiency argument disappears, though the attribution argument would remain.

Decisive Checks

The project succeeds if three things hold together: grouped typed attention beats the dense multi-task ablation on single-group removal selectivity; it retains measurably more DGM$^4$ performance after sequential AMG training; and it stays within noise of MIAN on the social-media benchmarks at lower measured latency. If only the efficiency result holds, we will report grouping as a compression technique for this domain and nothing more. If the random-assignment ablation matches the typed model, we conclude that the type supervision, not the grouping structure, carries the effect.