Much of the misinformation on social media is "out-of-context": a real, unedited photo is paired with a caption that misstates who, where, when, or what. Neither the image nor the text looks forged on its own. A detector therefore has to judge whether the two agree, and current detectors are weakest when the caption names the right kind of entity in the wrong context. Data augmentation is the usual remedy for scarce, hard training cases, but multimodal augmentation was designed to preserve image–text agreement. In this task, agreement is the label. We propose to turn augmentation into a label-controlled instrument. Each real news image–caption pair yields a family of training examples made with one shared rewriting process. One kind changes the caption or image together so the pair still agrees. One kind changes a single attribute so the image no longer supports the caption. One kind replaces a specific image with a generic one, so the caption is neither supported nor refuted. A compact vision–language encoder is then given continued pre-training on in-domain news pairs, with these minimal counterfactuals as its contrastive negatives.
The two-sentence version is this. Out-of-context detectors fail on near-miss pairs, and the negatives they train on (retrieved mismatched captions) are rarely near misses. We generate minimal, typed caption–image edits in matched positive/negative pairs, so that the only thing separating "true" from "false" is the cross-modal fact. We then use them to specialize a small encoder to the news domain, testing whether this beats both domain pre-training alone and much larger generalist models.
Core Claim
The claim is that where the negatives come from matters more than how many there are, and more than the backbone's scale. Take a compact CLIP-style encoder: an image encoder and a text encoder trained so matching pairs score high. If it is continued-pre-trained on news pairs whose negatives differ from positives by exactly one typed attribute (person, location, date, event type, quantity), it should close most of the gap on the failure modes the literature documents. Those include the Person Match split of NewsCLIPpings and the generic-image Support_Text errors in Findings of Factify 2. This should hold at a matched number of negatives against retrieval-based negatives, and against domain continued pre-training without counterfactuals.
The design rule that makes this more than "generate hard negatives" is synchronized artifacts. Positives pass through the same rewriting and image-editing machinery as negatives. Rewriting, inpainting, or flipping therefore never correlates with the label, and the only learnable signal is the cross-modal fact.
Where This Sits
Three bodies of work supply the pieces.
The synchronized-augmentation literature supplies the principle that augmenting one modality without coordinating the other corrupts labels. LeMDA names this "semantic disharmony" and avoids it with a learned feature-space augmenter, reporting roughly 6% accuracy gains on Hateful Memes and Petfinder over standard multimodal networks. Zang et al. show that flipping an image requires swapping "left" and "right" in the referring text, and Wu et al. co-edit text attributes and pixels using a language model plus inpainting. We reuse the co-editing machinery of Wu et al. and the flip-with-text-swap rule of Zang et al. as our label-preserving edits. Where these methods aim to keep the label fixed, our contribution is to also produce controlled label flips from the same edit process. LeMDA's latent perturbations cannot target a named attribute, and so cannot produce a "same scene, wrong person" negative.
Misinformation work motivates the gap and constrains the design. NewsCLIPpings builds its falsified pairs by retrieving other real captions using CLIP similarity, person match, and scene match. Models in that study sit between 60% and 70% accuracy, near untrained humans at 65.6%, and struggle most when correct entities appear in incorrect contexts. Retrieved captions differ from the true one in many ways at once, so a model can learn coarse topic mismatch instead of the specific contradiction.
Shalabi et al. generate synthetic counterparts with BLIP-2 and Stable Diffusion, but only at test time, as a "consistency mirror" (68.0% on NewsCLIPpings). Their own analysis notes that a generated generic politician still matches a caption about a specific one. That is exactly why our edits are entity-typed rather than free generation. CMA "augments" by extracting five feature views from frozen CLIP for few-shot detection. No new image–text evidence is created, and the encoder never changes.
Sun et al. and FineFake separate cross-modal inconsistency from contradiction with external knowledge. Our scope is deliberately the first kind: we test whether the image supports the caption, not whether the caption is true of the world. Fact or Fake? reports that pixel-forensics detectors miss claim-level inconsistency. This constrains us: image-side edits must not leave forensic traces that correlate with the label, which is the synchronized-artifacts rule again.
The domain-specialization and domain pre-training work supplies the training regime but also needs caution. Männistö finds domain pre-training predicts downstream performance better than the choice of parameter-efficient fine-tuning method. MonoCoder and BERTHop show that compact in-domain models can beat larger diluted ones. These are code and clinical results, not news. They motivate testing domain continued pre-training of the encoder, which multi-domain detectors such as DAMMFND, MDFEND, and DPOD never do: they keep general BERT or CLIP frozen or lightly prompted. They do not prove it will help here. We therefore treat "domain continued pre-training alone" as a serious competitor, not a component we assume works.
Research Program
The edit generator. We start from VisualNews-style image–caption pairs, the source of NewsCLIPpings, after excluding every image and caption in any evaluation split. Each caption is parsed into typed slots using named-entity recognition and, where possible, Wikidata linking as in FineFake: people and their roles, locations, dates, event types, and counts. Slot vocabularies are built per news domain (politics, disasters, health, conflict, entertainment), because the plausible substitutes differ by domain. A plausible wrong location for a flood photo is another flood-prone city; a plausible wrong person for a summit photo is another head of state. This is where domain specialization enters: the edit schema and substitute pools are domain-specific, so the negatives look like realistic local misinformation rather than random swaps.
From each pair, a single instruction-following language model produces three outputs:
- A consistent rewrite: a paraphrase that keeps all slot values, sometimes paired with a synchronized image edit such as a horizontal flip with left/right swap.
- A contradicted rewrite: exactly one slot replaced with a same-domain, same-type substitute that the image cannot support, such as a different visible person or a location whose landmarks are absent.
- An underdetermined version: the original caption paired with a generic image of the same scene type, retrieved from a different event.
The contradicted rewrite also has an image-side variant. The caption is kept, and the image is swapped for one depicting the substituted entity in a similar scene. In a smaller arm, the image is inpainted following Wu et al., with positives inpainted on irrelevant regions so that inpainting traces carry no label information.
The underdetermined class matters because it mirrors FACTIFY's Insufficient categories. It targets the Factify 2 failure in which generic drone or school images paired with authentic text are misread. A model trained only on consistent versus contradicted pairs would wrongly learn that "generic image" means "false."
The training objective. Let $s(I,C)$ be the encoder's similarity between image $I$ and caption $C$. Write $C^{+}$ for a consistent rewrite, $C^{\varnothing}$ for the caption under an underdetermined pairing, and $C^{-}$ for a contradicted rewrite. On top of the standard in-batch contrastive loss used for continued pre-training, we add an ordered margin loss:
$$ \mathcal{L}_{\text{order}} = \max\big(0,\ m_1 - s(I,C^{+}) + s(I,C^{\varnothing})\big) + \max\big(0,\ m_2 - s(I,C^{\varnothing}) + s(I,C^{-})\big) $$
Here $m_1$ and $m_2$ are margins with $m_1 < m_2$. The loss encodes the intended ordering: supported above underdetermined, and underdetermined well above contradicted. The image-side variants enter the same loss with roles swapped, holding the caption fixed and varying the image. After continued pre-training, a light classification head is fine-tuned per benchmark, using that benchmark's own label space.
The compact model. The backbone is CLIP ViT-B/32 as the primary model, to match the NewsCLIPpings baselines, and ViT-L/14 as a scale check. It is small enough that continued pre-training on a few million in-domain pairs fits a modest compute budget. That same budget is given to every control arm.
Baselines
The naive reference point is off-the-shelf CLIP ViT-B/32 fine-tuned on NewsCLIPpings with its native retrieved negatives. Luo et al. report this family between 60% and 70%, with their baseline quoted at 65.9% accuracy in Shalabi et al.. We rerun it ourselves under our tuning budget, so the comparison does not rest on quoted numbers.
The strongest NewsCLIPpings numbers in our file are Shalabi et al.'s consistency mirror at 68.0% accuracy and the Abdelnabi et al. baseline at 66.1% accuracy (as quoted by Shalabi et al.). DPOD is described as outperforming CLIP-FT and others by a significant margin, but no reported result was found on Lacuna for DPOD's NewsCLIPpings accuracy. We will therefore reimplement DPOD's domain prompt tuning on our backbone and report it ourselves; it is the most direct domain-specialization competitor.
On Factify 2, the bar is Triple-Check's 81.82% weighted-average F1 (Findings of Factify 2). That system uses DeBERTa, Swinv2, and CLIP with adapters, so it is far larger than our encoder. Beating it outright is not required, but matching it within two points with one compact encoder, while improving the Support_Text and Insufficient classes specifically, would count as a win.
On MMFakeBench, the generalist ceiling is GPT-4V with the MMD-Agent reasoning loop at 61.6% F1 (MMFakeBench). We compare on the cross-modal consistency distortion subset, where our method is meant to apply, and report the full benchmark honestly. On FineFake, KEAN's 0.797 binary F1 (FineFake) is the reference, and we break results out on its text-image inconsistency class. For VERITE, no reported result was found on Lacuna, so it serves purely as a transfer test against our own reruns.
Four ablations isolate the key ingredients, all with identical data sources, number of negatives, compute, and hyperparameter search:
- Domain pre-training without counterfactuals. Continued pre-training on the same in-domain pairs with only in-batch negatives.
- Retrieved negatives. Continued pre-training with NewsCLIPpings-style retrieved negatives in place of our edits, at a matched count.
- No artifact synchronization. Our edits with positives left unrewritten, which tests the rule itself.
- Domain-agnostic schema. Substitutes drawn from all domains rather than the same domain.
LeMDA, applied to the same backbone, is the strongest synchronized-augmentation competitor and runs under the same budget.
Why not just…?
Why not just fine-tune CLIP on NewsCLIPpings, which already contains hard negatives? Its negatives are retrieved real captions, which differ from the true caption in topic, style, and several slots at once. Luo et al.'s finding that models still fail on Person Match suggests this signal does not teach slot-level contradiction. The matched-count comparison between retrieved and minimal-edit negatives settles this directly. If retrieved negatives match ours on the Person Match split, the core claim is wrong.
Why not just do domain continued pre-training and skip the edits? This is the most serious threat. Männistö and MonoCoder suggest that domain pre-training alone often captures most of the gain. If the in-batch-only arm closes most of the gap to our full method, the honest conclusion is that the counterfactuals add little, and we will report it that way. We expect the difference to concentrate on Person Match and on Insufficient versus Refute, so we report per-split results rather than only overall averages, where such a difference could hide.
Why not just use LeMDA-style learned latent augmentation? LeMDA is built to keep the label fixed under perturbation, and it beat MixGen in its own evaluation. Its perturbations are untyped vectors, though, so they cannot express "same scene, different head of state." It was also never tested on out-of-context news. We run it on the identical backbone and data. If it matches our method, typed input-space edits are unnecessary.
Why not just use a frontier vision–language model with tools? On MMFakeBench, GPT-4V reaches about 51% F1 with standard prompting and 61.6% with MMD-Agent. MMD-Agent depends on Wikipedia lookups that do not exist for breaking events. A compact encoder is also cheaper to run on a high-volume stream. We do not claim superiority on textual-veracity or visual-forgery subsets, which our method does not target. We report the cross-modal subset and the full benchmark separately, so that the scope of any advantage is visible.
Proposed Experiments and Decisive Checks
Training uses VisualNews-derived pairs with all evaluation images removed, and we deduplicate against benchmark images by perceptual hashing. Evaluation covers NewsCLIPpings (overall and per retrieval heuristic), VERITE (transfer without further training), Factify 2 and FACTIFY (five-way entailment with per-class F1), FineFake (per topic and per fine-grained class), and the MMFakeBench cross-modal subset.
Three checks decide whether the idea holds.
The first is the artifact probe. A text-only classifier, given only rewritten captions with no image, must fail to separate consistent from contradicted rewrites, scoring near chance. An image-only probe must likewise fail to separate inpainted positives from negatives. If either probe succeeds, the model may be learning edit traces, and the gains are suspect. The unsynchronized ablation should show those probes succeeding, which also confirms the probes are sensitive enough to catch the problem.
The second is the Person Match and Scene Match breakdown on NewsCLIPpings. The claim predicts the largest gains on these splits relative to both the retrieved-negatives and domain-pre-training-only arms. Uniform small gains across all splits would instead point to a generic data-volume effect.
The third is the underdetermined class test. Removing $C^{\varnothing}$ from training should raise false "Refute" predictions on Factify 2 Support_Text and Insufficient items with generic images. If it does not, the three-way ordering adds nothing over a binary objective.
A final check probes forgetting, which MMFakeBench reports for specialized detectors such as HAMMER. We fine-tune the specialized encoder sequentially on NewsCLIPpings and then Factify 2, and measure retained NewsCLIPpings accuracy against the plain fine-tuned CLIP. The hypothesis is that an encoder whose representation already orders support, underdetermination, and contradiction needs a smaller head update, and so forgets less. The experiment reports retained accuracy whichever way it comes out.
References
- NewsCLIPpings: Automatic Generation of Out-of-Context Multimodal Media
- Findings of factify 2: multimodal fake news detection
- Learning Multimodal Data Augmentation in Feature Space
- RESMatch: Referring expression segmentation in a semi-supervised manner
- Towards reporting bias in visual-language datasets: bimodal augmentation by decoupling object-attribute association
- Image-Text Out-Of-Context Detection Using Synthetic Multimodal Misinformation
- Cross-modal augmentation for few-shot multimodal fake news detection
- Inconsistency Matters: A Knowledge-guided Dual-inconsistency Network for Multi-modal Rumor Detection
- FineFake: A Knowledge-Enriched Dataset for Fine-Grained Multi-Domain Fake News Detection
- Fact or Fake? Assessing the Role of Deepfake Detectors in Multimodal Misinformation Detection
- A Comparative Study of PEFT Methods for Python Code Generation
- MonoCoder: Domain-Specific Code Language Model for HPC Codes and Tasks
- BERTHop: An Effective Vision-and-Language Model for Chest X-ray Disease Diagnosis
- DAMMFND: Domain-Aware Multimodal Multi-view Fake News Detection
- MDFEND: Multi-domain Fake News Detection
- Can Out-of-Domain Data Help to Learn Domain-Specific Prompts for Multimodal Misinformation Detection?
- MMFakeBench: A Mixed-Source Multimodal Misinformation Detection Benchmark for LVLMs