Counterfactual Evidence Labels for Long Multimodal Documents

Benchmarks for question answering over long PDFs, such as reports, manuals and filings with charts and tables, grade the retrieval step against a few "gold" pages that an annotator marked as containing the answer. In a 100-page report the same number often appears in a summary table, a chart and a paragraph, so a retriever that finds a different but equally sufficient page is scored as wrong. This proposal replaces "the page the annotator marked" with a testable definition of relevance. A set of pages counts as evidence if a frozen reader model answers correctly when given those pages, and fails when given nothing. Candidate pages are then checked by swapping them into the gold set in place of a gold page.

The claim is that this counterfactual, set-level test recovers the missing positive labels on MMLongBench-Doc and LongDocURL more accurately than asking a model to judge relevance. It is also meant to be robust to a known failure of vision-language models, which answer from language priors instead of the pages. If the corrected labels reorder the leaderboard of page retrievers relative to a small exhaustive human audit, then current retrieval numbers on long multimodal documents are partly measuring agreement with annotators. The deliverables are corrected evaluation labels and a check of whether a small set of verified training labels beats a larger noisy one.

Core Claim

Relevance label sparsity, meaning test collections where most truly relevant items were never judged, is well documented for web passages and image captions. Overview of the TREC 2020 Deep Learning Track shows leaderboards built from one or a few positives per query diverge from dense judgments. ECCV Caption found up to 8.5 times more valid caption-to-image matches than MS-COCO records, and a Kendall's τ of only 0.47 between old and corrected rankings. Neither study touched multi-page visual documents, where the problem should be worse for two reasons.

The first reason is fragmentation. About a third of MMLongBench-Doc questions need evidence from several pages (MMLONGBENCH-DOC), so relevance belongs to page sets, not to single pages. The second reason is that page retrieval is now the bottleneck being optimized. TRIDENT builds per-page text records containing figure captions, section paths and entity tags, and uses them to select a variable number of pages. It raises ColPali F1@5 on MMLongBench-Doc from 0.332 to 0.581, all measured against fixed gold pages. An adaptive selector that returns an unmarked page with the same table is penalized for it.

We define a page set $S$ as sufficient evidence for question $q$ with reference answer $a$ under reader $R$ when

$$\mathrm{Suff}_R(S) = \mathbb{1}[R(q,S)=a]\cdot\mathbb{1}[R(q,\varnothing)\neq a],$$

where $R(q,S)$ is the reader's answer given only the page images in $S$ and $R(q,\varnothing)$ is its answer with no pages. The second factor is the guard against linguistic shortcuts. Plot Twist shows multimodal models can answer hard chart questions from textual priors while missing basic chart details. A question the reader answers blind therefore cannot certify any page, and we report it separately instead of labeling it.

A candidate page $c$ is labeled substitutable evidence if some gold page $g$ in the gold set $G$ can be swapped for it, that is, $\mathrm{Suff}_R(G\setminus\{g\}\cup\{c\})=1$ while $\mathrm{Suff}_R(G)=1$. The gold set acts as an anchor. This borrows the central idea of One-Shot Labeling, which judges unlabeled passages relative to a single known relevant one and reaches system-ranking Kendall's τ above 0.90 even though individual label F1 is only 0.55–0.59. We change the comparison from "is this passage similar to the known positive?" to "does this page do the known positive's job in answering?". That change is what lets the labels respect the cross-page structure of the question.

Research Program

The work has three parts.

Part 1: candidate nomination. For every answerable question in MMLongBench-Doc and LongDocURL, we pool the top pages from a diverse set of retrievers. The pool includes ColPali (ColPali), the ColQwen2.5 models that lead ViDoRe V2, BGE-M3 over OCR text, and TRIDENT-R on top of each. This follows the ensemble nomination that made ECCV Caption tractable. Each nominated page receives the substitution test against each gold page. For single-page questions this is one reader call per candidate. For cross-page questions it is one call per candidate–gold pair. The total stays within tens of thousands of calls.

Part 2: countering algorithmic echo. A labeler that shares design choices with the systems it grades will favor those systems. To limit this, we run the sufficiency test with three readers from different model families. We then report labels under each reader, under majority vote and under unanimity. If the leaderboard changes only under one reader family, that change is treated as an artifact.

Part 3: validation and the data-quality question. We commission exhaustive page-by-page human judgment for a stratified sample of about 150 MMLongBench-Doc questions. The sample is stratified by domain, by single- versus cross-page questions, and by evidence type (chart, table, text, layout). Annotators judge sufficiency, not topical relevance. This sample is the reference against which every labeling method is scored.

We then ask the question from the quality–quantity literature: is a small set of verified labels a better training signal than a larger set of sparse, noisy ones? Earlier results only point this way. SHED shows 6% of an instruction set beating the full set. Experimental Design SFT shows 45k selected prompts matching 90k random ones. Contrastive Confidence Regularization shows that filtering false negatives lets 5 hard negatives beat 15. All three are text-only, and none uses labels verified by counterfactual answering. We fine-tune a ColPali-style reranker on relabeled LongDocURL and evaluate on the human-audited MMLongBench-Doc subset, so training and test documents are disjoint.

Baselines

Original gold pages (do nothing). This is the naive reference point. The benchmark's own page annotations are scored with F1@5 and nDCG@5. Every other labeling method is compared with it in two ways: agreement with the human audit, and Kendall's τ between the system ranking it induces and the human-audit ranking.

Unanchored VLM judge. This is the strongest competing way to fill label holes. A strong vision-language model is shown the question and one page and asked whether the page contains the answer, the standard LLM-as-a-judge move. It gets the same readers, the same candidate pool and the same prompt-tuning budget, with prompts tuned on a held-out 20 questions. Our method beats it if it achieves higher precision on added positives against the human audit at equal or better recall, and a system-ranking τ closer to the human ranking.

Anchored similarity judge. This is a direct multimodal port of 1SL's DuoPrompt, which asks whether the candidate page is as relevant as the gold page, without an answering step. It isolates whether the gain comes from anchoring or from counterfactual answering.

Strongest retrieval system under evaluation. This is TRIDENT-R. Its reported ColPali F1@5 on MMLongBench-Doc is 0.581, versus 0.332 without it (TRIDENT). We do not propose to beat TRIDENT-R. We test whether its measured margin over non-adaptive retrievers grows, shrinks or reorders under corrected labels, and the prediction is that the margin grows because adaptive selection surfaces substitutes. For the training experiment, the strongest competitor is the same reranker trained on original labels with Passage Sieve-style false-negative filtering. That work reports MS MARCO MRR@10 rising from 39.5 to 39.8. We found no reported result on Lacuna for it on multimodal documents.

Ablations. Three ablations isolate the key ingredients: removing the blind-reader factor, testing a candidate alone instead of by substitution into the gold set, and using a single reader instead of three families.

Why not just…?

Why not just ask a strong VLM whether each page is relevant? This is the real threat. Zero-shot relevance judgment may match human sufficiency judgments well on single-page questions. Two things count against it. The file's evidence is that unanchored judges drift across domains, while anchored judges cut 1SL's false-positive rate on TREC 2019 from 85.7% to 0%. Plot Twist shows these models reason from priors about charts, and a page-in-isolation judgment cannot tell "mentions the topic" from "supplies the number." The direct comparison against the human audit settles the question. If the plain judge matches us on cross-page and chart questions, the counterfactual machinery is unnecessary and we will say so.

Why not just evaluate end-to-end answer accuracy and ignore page labels? Answer accuracy mixes retrieval with reading. AWM found that 42.5% of correctly answered MMLongBench-Doc questions could not be answered from the agent's own notes, so correct answers do not certify grounded retrieval. We still report QA accuracy for every system, and we identify the questions where corrected retrieval labels and answer accuracy disagree.

Why not just re-annotate densely by hand? On the 150-question audit we do exactly that. Doing it for every question on every benchmark does not scale to new document collections. The value of the automated labels depends entirely on matching the audit, which is why the audit is the reference and not an afterthought.

Why not skip retrieval with a long-context model? MMLongBench-Doc reports large gains when models receive only the relevant pages instead of the full document, over 20 points for Gemini-1.5-Pro. Page selection therefore matters, and so do the labels used to measure it.

Proposed Experiments

The decisive check is a comparison of system rankings. We take roughly a dozen retrieval configurations: four retrievers, each with and without TRIDENT-R, plus fixed and adaptive page counts. We rank them under four label sets: original labels, the unanchored judge, the anchored similarity judge, and counterfactual substitution labels. We then compute Kendall's τ of each ranking against the ranking induced by the exhaustive human audit, broken down by single- versus cross-page questions and by evidence type.

The proposal succeeds if three things hold together. Original labels must correlate noticeably below 1 with the audit ranking. Counterfactual labels must correlate highest. The advantage must hold under majority and unanimous readers and not only under one reader family. It fails if the unanchored judge matches it, or if the leaderboard does not move.

We also measure label precision and recall against the audit, the share of questions excluded by the blind-reader guard (reported per domain), and the change in TRIDENT-R's margin. The secondary experiment trains the reranker on a few thousand verified LongDocURL labels versus all original labels with and without false-negative filtering. It tests whether the corrected labels also make a better small training set, and judges each reranker on the audited MMLongBench-Doc subset under both label sets. A reranker that only looks better under our labels is not counted as better.