When a misleading claim comes with a chart, a table screenshot or a photo, the fact that settles it is often tiny: a relabeled axis tick, a truncated y-axis, a legend swapped between two lines, a street sign in the background. Current multimodal fact-checkers handle such images in one of two ways. Some describe the whole image in words before reasoning about it. Others send the whole image to a large proprietary vision-language model and ask for a verdict and a paragraph of justification. Neither can say where in the image the claim breaks, and both miss details that a whole-image view never resolves. On the financial chart benchmark FIN-FACT, the best proprietary vision model reaches only 0.58 accuracy, and the authors report hallucinated watermarks and misread axis labels (Fin-Fact).
We propose a verifier that learns to point. It splits a claim into small sub-claims. For each sub-claim it looks at the image at several zoom levels and learns which zoom level and location to trust. It then combines the per-sub-claim verdicts with a fixed logical rule: the claim is supported only if every part is supported, and refuted if any part is refuted. The whole system trains from claim-level true/false labels alone. As a by-product of that rule, the model has to name a "culprit": the sub-claim and the image region, at the right zoom, that refutes the claim. That culprit is the explanation, and we can test whether it is faithful by removing the region and checking whether the verdict changes.
Core Claim
Our claim is that phrase-level culprit finding for text, as done by LOREN, can be carried over to images. LOREN gives each claim phrase a three-valued hidden verdict (supported, refuted, not enough info). It trains those hidden verdicts from claim labels through a differentiable version of AND/OR logic, and reports 74.3 F1 at identifying the phrases that refute a claim. The visual counterpart needs one ingredient LOREN never had: a choice over scale. The decisive evidence for "revenue fell 5% in Q3" is a small crop around one bar and its tick label. For "the chart shows steady growth" it is the whole plot. We therefore make multi-scale spatial attention the mechanism that selects each sub-claim's evidence. Multi-scale spatial attention means weighting several views of the image at different receptive-field sizes and fusing them per location. It is the same idea as the gating over parallel scales in AFCN's multiscale attention, dilated pyramids in ACNet, and window-restricted heads in Multi-Scale Self-Attention. Here it is conditioned on the sub-claim and supervised only through the logic.
The contribution is not a better vision backbone. It is a trainable, open-weights verifier whose explanations are image regions with a known scale, and whose faithfulness can be measured directly. We expect three things. The model should beat compute-matched uniform high-resolution tiling specifically on claims decided by small visual elements. It should match whole-image fine-tuning elsewhere. It should produce region explanations that pass deletion tests far more often than attention heatmaps extracted after the fact.
Research Program
The system has four parts.
The first is a claim decomposer, built by prompting an instruction-tuned open model to write atomic sub-claims, in the spirit of QABriefer and QA-NatVer. Decomposition Dilemmas warns that decomposition can hurt as well as help. We treat that as a constraint and include a no-decomposition ablation. We also cap sub-claims at a small number.
The second is a crop pyramid: the full image, a 2×2 grid, a 4×4 grid, and optionally boxes from an off-the-shelf text detector, since charts put most of their facts in text. The third is an open vision-language model, a LLaVA-class backbone as used by How to Train Your Fact Verifier. For each sub-claim $k$ and crop $r$ it outputs a verdict distribution $p_{k,r}$ over {supported, refuted, NEI}. The fourth is a small scoring head that produces attention weights $a_{k,r}$. These are a softmax over crops, computed from the sub-claim embedding and each crop's feature and scale embedding. The sub-claim verdict is $p_k=\sum_r a_{k,r}\,p_{k,r}$, and the claim-level probabilities follow LOREN's soft logic:
$$P(\text{SUP}) = \prod_k p_k(\text{SUP}), \qquad P(\text{REF}) = 1-\prod_k \big(1-p_k(\text{REF})\big),$$
with NEI taking the remaining probability mass. Training minimizes cross-entropy between these claim-level probabilities and the gold label. We add LOREN's teacher-student distillation so that the per-sub-claim verdicts stay consistent with the logic rather than drifting. The explanation is the sub-claim with the highest refutation probability, paired with its highest-weighted crop.
Scoring every crop for every sub-claim is expensive: 21 crops times $K$ sub-claims. We therefore also build a coarse-to-fine variant. It scores the 1×1 and 2×2 levels, then expands only crops whose attention weight passes a threshold. Every comparison below reports forward passes per claim so that compute can be matched.
We also build a controlled evaluation set, since no existing benchmark in our file annotates visual culprits. Starting from FIN-FACT charts and figures from VeriSciQA, we write claims whose truth depends on one chart element. We then programmatically perturb exactly that element (a tick label, a bar height, a legend entry, an axis range), which flips the gold label. The claim text stays identical, so claim-only shortcuts cannot solve the pairs. This addresses the worry in Fact-checking or gut-checking? that verifiers exploit surface cues. It matters here because on FIN-FACT, text-only GPT-4 (F1 0.76) outperforms GPT-4V. Each perturbed pair also comes with the ground-truth region and its area, which lets us stratify results by element size. Separately, we will hand-annotate culprit regions on a few hundred refuted FIN-FACT claims, mirroring LOREN's manual culprit set.
Baselines
The naive reference points are the majority label and a claim-only text model fine-tuned on the same training data. The claim-only model tells us how much of each benchmark can be solved without looking at the image. On the perturbed pairs it should sit at chance by construction.
The strongest reported competitor on the core benchmark is a proprietary zero-shot vision-language model. Gemini Vision Pro reaches 0.58 accuracy on FIN-FACT multimodal verification (Fin-Fact). We will rerun it through the same API on our splits and on the perturbed set, prompting it to name the image region it relied on. Beating it means higher accuracy on the FIN-FACT test split and higher region-pointing accuracy. We do not claim the comparison is compute-matched, and we will say so.
The strongest open competitor is the same backbone fine-tuned on whole images, with GPT-4o-generated explanations as extra supervision. This is the recipe that took LLaVA to 65.07 F1 on MOCHEG in How to Train Your Fact Verifier. It gets the same data, the same number of tuning trials, and the same GPU budget. Its explanations are gradient-weighted attention maps plus generated text. Beating it means equal or better F1 on MOCHEG and FIN-FACT, and clearly better performance on small-element perturbations.
The ablation that isolates our key ingredient is uniform tiling. This baseline feeds the identical crop pyramid through the identical backbone, but averages crop features (or concatenates them, as high-resolution "any-resolution" encoders do) and uses a learned classifier head instead of the sub-claim-conditioned scale attention and the logic. It gets the same number of forward passes. Two further ablations remove the logic (a learned MLP combines the sub-claim verdicts) and the decomposition (one sub-claim equals the whole claim). We will count the scale-attention-plus-logic design as justified only if it beats uniform tiling on the perturbed set at matched compute, with the gain concentrated on the smallest-area elements.
For out-of-context photos we will report MAD-Sherlock's 85.2% accuracy on VERITE only as a reference. That system uses web retrieval and GPT-4o debate, which our model does not.
Why not just…?
Why not just textualize the image and let a strong LLM reason? MAFT captions images before reasoning, so any detail the captioner omits is gone for good, and the output cannot point to a region. We will test this directly rather than assume it. A MAFT-style pipeline (open captioner followed by the same open LLM) runs on the perturbed set, where we know which element flips the label. If captions capture tick-level edits as well as our crops do, the spatial machinery is unnecessary for accuracy, and we will say so. Region grounding would then be the only remaining advantage.
Why not just prompt a frontier multimodal model or an agent like DEFAME? DEFAME reaches 69.7% on post-cutoff ClaimReview2024+, and it would beat us there, because its gains come from web retrieval, which we do not address. On evidence contained in the image, frontier models are weak on charts (0.58 on FIN-FACT) and are reported to hallucinate visual details. Their stated regions are also unverifiable self-reports. The deletion test below measures whether those self-reports are faithful. We treat DEFAME as complementary: our verifier could serve as its image-inspection tool.
Why not just raise the input resolution or tile the image uniformly? This is the real threat. Much of any gain from "multi-scale attention" could come simply from showing the model more pixels. That is exactly why the uniform-tiling ablation uses the same crops and the same compute. If it matches us on accuracy, the accuracy claim fails. What remains to test is whether logic-supervised scale selection gives more faithful culprits than tiling combined with post-hoc heatmaps. We will report the two outcomes separately and not blend them.
Why not just use attention maps or Grad-CAM from a fine-tuned model as the explanation? Such maps are computed after the fact, and nothing forces the verdict to depend on the highlighted region. In our model the verdict is a weighted sum of region verdicts, so deleting a region has a predictable effect. The deletion test measures whether this design advantage is real.
Proposed Experiments
The decisive experiment is the perturbed chart set, with results stratified by the area of the perturbed element. The prediction is specific. Our model and uniform tiling should be similar on large elements. Our model should lead on elements covering under about 1% of the image. Whole-image fine-tuning should approach chance on those elements, since a 336-pixel encoder cannot resolve them.
On the same set, we measure how often the predicted culprit crop contains the perturbed element, and whether the selected scale matches its size. We then run faithfulness tests. Masking the predicted culprit region should flip or neutralize the verdict. Masking a random region of equal area should not. Our method is compared on both counts against Grad-CAM from the fine-tuned baseline and against regions named by the proprietary model. We also report LOREN's agreement metric: how often the claim verdict equals the logical combination of sub-claim verdicts.
On natural data, we report accuracy and macro-F1 on FIN-FACT, F1 on MOCHEG against the 65.07 figure, and culprit-pointing accuracy on the hand-annotated FIN-FACT subset. As a transfer check without retrieval, we report VERITE accuracy. We expect to trail MAD-Sherlock there. We also report forward passes and wall-clock time per claim for the full and coarse-to-fine variants.
The proposal fails cleanly in three cases. If uniform tiling matches us on small elements at equal compute, the accuracy claim is gone. If our culprits pass deletion tests no more often than Grad-CAM, the explainability claim is gone. If the logic ablation equals the full model on both, the logic adds nothing. Each outcome would still be informative about where fine visual evidence enters multimodal fact-checking.
References
- Fin-Fact: A Benchmark Dataset for Multimodal Financial Fact Checking and Explanation Generation
- LOREN: Logic-Regularized Reasoning for Interpretable Fact Verification
- A Novel Attention Fully Convolutional Network Method for Synthetic Aperture Radar Image Segmentation
- Attention Convolutional Binary Neural Tree for Fine-Grained Visual Categorization
- Multi-Scale Self-Attention for Text Classification
- Generating Fact Checking Briefs
- QA-NatVer: Question Answering for Natural Logic-based Fact Verification
- Decomposition Dilemmas: Does Claim Decomposition Boost or Burden Fact-Checking Performance?
- How to Train Your Fact Verifier: Knowledge Transfer with Multimodal Open Models
- Automatic Fake News Detection: Are current models “fact-checking” or“gut-checking”?
- MAD-Sherlock: Multi-Agent Debate for Visual Misinformation Detection
- MAFT: Multimodal Automated Fact-Checking via Textualization
- DEFAME: Dynamic Evidence-based FAct-checking with Multimodal Experts