Counterfactual-Checked Visual Compression for Full-Text Clinical Trial Evidence

Systems that check medical claims usually read only abstracts, and abstracts leave out a great deal. Nearly half of known protein interactions appear only in the body of papers (PreBIND and Textomy). Trial abstracts routinely omit randomization and blinding details (Speich et al.). The deciding numbers (doses, eligibility thresholds, adverse-event counts) usually sit in tables and fine print. Reading full documents as page images avoids brittle text extraction, but vision-language models turn each page into hundreds or thousands of image tokens, so long documents quickly become too expensive. Methods that compress these tokens are judged on generic question-answering accuracy. That score cannot tell whether a compressed page still shows that a threshold was "≥ 18" rather than "≥ 16," or that an adverse event occurred in 3 patients rather than 8.

This note proposes a way to measure that loss and a way to avoid it. The measurement: we edit one decision-critical number or negation on a rendered trial page and require the model's verdict to flip. A model whose verdict does not flip never read the evidence. The method is an "anytime" zoom procedure. The model first skims every page at a tiny token budget, then re-renders only chosen tiles at higher resolution over a variable number of rounds. The round schedule is calibrated on about 20 examples against the flip metric rather than fixed in advance. We borrow that calibration idea from work on pruning diffusion sampling steps, which found that the right step schedule is data-dependent and that extra steps can hurt.

Core Claim

Our first claim is that generic visual-token compressors lose clinically decisive quantities far faster than their accuracy scores reveal. On clinical trial reports this effect is especially hidden, because verdict accuracy is already inflated by shortcuts: on NLI4CT, a BERT model that sees only the statement and never the trial report reaches 0.626 F1, and the majority-class baseline reaches 0.667 F1 (NLI4CT). Our second claim is that a multi-round zoom procedure, with its schedule tuned few-shot to the counterfactual metric, can keep near full-resolution evidence sensitivity at a small fraction of the image tokens. We also expect it to beat a fixed two-stage scheme when the needed evidence is scattered across several sections of the report.

Relation to Prior Work

Four groups of prior work bear on this proposal, in different roles.

Two papers motivate the need for full text. NLI4CT and its shared task (SemEval-2023 Task 7) pose claims against full Clinical Trial Reports, split into eligibility, intervention, results and adverse-event sections. They document a "numerical wall" on dosage and unit reasoning, but the benchmark is text-only. MultiVerS shows that encoding the whole document helps claim verification: the drop on context-dependent claims shrinks from 22.8% to 14%. It works on short documents, though, and never touches page images.

Four papers supply the compression methods we compare against. DART removes near-duplicate tokens and discards 88.9% of tokens while keeping 98.8% of LLaVA-1.5 performance. FEATHER fixes a bias in attention-based pruning, which tends to throw away the top of the image. CARES picks the input resolution before encoding. Correlation-guided document compression (Zhang et al.) strips whitespace. All four are validated on generic benchmarks, and CARES explicitly names dense, fine-cue domains as unresolved. None is evaluated against a domain metric.

The closest existing method is "Identify First, Reason Later" (A Table Is Worth 64 Tokens). It renders every table at 64 tokens, picks the relevant tables, and re-renders only those at 1,024 tokens. That saves 41% of tokens and gains 7 points on FinLongDocQA. It also documents "reasoning inflation": when a model sees images that are too compressed, it writes longer, more confused reasoning. We differ in four ways. We do not assume table boundaries are given; we zoom on a grid of tiles. We allow a variable number of rounds rather than exactly two. We set the schedule by calibrating against a clinical metric. And we evaluate on clinical trials, with counterfactual edits, rather than on finance.

Direction 1 supplies the calibration lens. Few-shot Temporal Pruning chooses which diffusion sampling steps to keep using Gaussian-process optimization over 10–20 validation examples. It shows that extra steps can destroy quality: BLEU fell from 27.80 at 4 steps to 4.50 at 50 steps on WMT16 English–Romanian. We treat zoom rounds the way that work treats sampling steps, as a schedule to search rather than a constant fixed in advance. The FTP collapse and reasoning inflation look like the same phenomenon in two settings, and we test that directly.

Finally, the counterfactual metric extends a lesson from radiology evaluation. Yu et al. showed that overlap metrics fail to penalize polarity-flipping errors, and they built entity-level metrics instead. We apply the same logic to evidence reading rather than to generated text.

Research Program

The first piece is a paired benchmark. We render each NLI4CT trial report to page images in two styles: a clean render, and a degraded, scan-like render with lower resolution and noise. Because we control the rendering, we know exactly which tile contains each gold evidence sentence. For each statement, a clinician-checked script makes a minimal edit to the decisive span in the image: a numeric change that crosses a stated threshold, a negation insertion, or a swapped arm label. It records whether that edit should flip entailment to contradiction or leave it unchanged.

The flip-consistency score is the fraction of edited pairs on which the model's verdict changes exactly when it should. The score equals 1 only if the model actually read the decisive span, and statement-only shortcuts score near chance. We will validate it by showing that the statement-only model scores near chance, and by comparing it against clinician judgments on a sample of pairs.

The second piece is the anytime zoom reader, built on an open vision-language model; Qwen2.5-VL is a natural choice since CARES reports results on it. In round 0, every page is shown at about 64 tokens and the model lists tiles likely to hold evidence. In each later round, the selected tiles are re-rendered at a higher budget and added to the context. The reader stops when its verdict and chosen tiles are unchanged across two consecutive rounds, or when a token cap is reached.

The schedule has four settings: per-round budget, tile size, maximum rounds and the stopping patience. We calibrate it with GP-Hedge over 20 training statements, maximizing flip-consistency minus a token penalty, following FTP's procedure. We will also study the resulting schedules directly: how many rounds each section type needs, whether results and adverse-event tables consistently need more rounds than eligibility text, and how generated reasoning length changes as budgets shrink.

Baselines

The do-nothing reference is full-resolution rendering of every page. It fixes the token cost we aim to cut and the flip-consistency ceiling we aim to approach.

The text upper bound is a text model reading the report text directly, with the same backbone in text mode. The strongest reported number on this task is @THiFLY's 0.856 F1 on NLI4CT Task 1 and 0.853 F1 on Task 2 (SemEval-2023 Task 7). That is a text system, and we report it for context only, not as a like-for-like target. The floors are the statement-only model (0.626 F1) and majority class (0.667 F1).

The strongest compression competitor is Identify First, Reason Later, reimplemented on our tile grid with its original 64/1,024 budgets. Its reported result is 41% token savings and +7 accuracy points on FinLongDocQA. There is no reported result on Lacuna for any visual compression method on NLI4CT or on clinical trial documents. We will also run DART, FEATHER, CARES and correlation-guided compression at matched average token counts.

For fairness, every method uses the same backbone, the same rendered pages and the same prompts. Every method with tunable settings gets the same 20-example calibration set and the same number of GP-Hedge evaluations. This means DART's keep ratio and the two-stage budgets are tuned too.

Beating a baseline means higher flip-consistency at equal or lower total tokens, counting both image and generated tokens. The gap must hold on held-out statements with bootstrap confidence intervals excluding zero. It must also come without losing Task 1 F1.

Two ablations isolate our ingredients. The first keeps the multi-round reader but calibrates it to plain F1 instead of flip-consistency; this isolates the effect of the metric. The second caps the reader at two rounds; this isolates the value of a variable schedule over a fixed one.

Why not just…?

Why not just use the text, or OCR the pages, and skip visual tokens? For NLI4CT the clean text exists, so the text route is a real threat on clean renders, and we report it as the upper bound. The question the visual route answers is what happens when the text layer is unreliable. We test this with the degraded, scan-like renders, running an OCR-then-text pipeline as an additional baseline. If OCR-plus-text matches the zoom reader on degraded pages too, we will report that the visual route is unnecessary for text-heavy trial reports. The contribution then narrows to the benchmark and the finding that compressors hide evidence loss.

Why not just use the two-stage identify-then-reason scheme as published? It may be sufficient when evidence sits in one section. We expect it to fail when a statement needs, say, both the eligibility threshold and the adverse-event table. NLI4CT's multi-evidence design makes such statements common. We will split results by the number of sections a statement needs. If two rounds match the variable schedule even on multi-section statements, the variable-round claim is refuted.

Why not just tune an aggressive generic compressor like DART to a smaller budget? DART's 98.8% retention was measured on generic benchmarks. Duplication-based pruning may treat rows of near-identical table cells as duplicates, and those cells differ only in the digits that matter. Matched-budget comparisons on the flip metric settle this directly.

Why not just report F1 and skip the new metric? F1 near 0.63 is achievable without reading the report at all. We will show whether F1 and flip-consistency rank the compressors differently. If they agree, the metric adds little and we will say so.

Proposed Experiments

The first decisive check tests whether accuracy hides the loss. Across compressors at 10–30% of full tokens, we compare the relative drop in Task 1 F1 with the relative drop in flip-consistency. The first claim holds only if flip-consistency falls substantially more than F1, most sharply for numeric edits in results and adverse-event tables.

The second decisive check tests the method. The calibrated anytime reader should reach flip-consistency within a few points of full resolution while using at most about 30% of the tokens. It should beat the tuned two-stage baseline on multi-section statements.

The third check links the two directions. We force extra rounds beyond the calibrated count and measure whether flip-consistency degrades and generated reasoning lengthens. That would be a document-reading analogue of FTP's step collapse. A null result would tell us the schedule matters only for cost, not for quality.

We also report Task 2 evidence F1 by comparing the selected tiles against the gold evidence tiles, and time to first token.