Verified Change Claims in Serial Chest X-ray Reports

When a radiologist reads a chest X-ray for a patient who has had earlier scans, much of the report compares the new scan with the old ones. Typical sentences are "the left effusion has worsened," "the right lower lobe opacity is improving," and "cardiomegaly is unchanged." Automated report generators now produce these comparative sentences routinely. They also invent them: a model may write "unchanged" as filler, or call an effusion "worse" when the images are ambiguous. No current system can tell a clinician what fraction of its stated changes are wrong. This note proposes a method that can. Each comparative sentence in a report becomes a small, structured claim (region, finding, direction of change). An image-only model that compares the current and prior scans scores each claim. The scores are turned into statistically calibrated p-values, and a false discovery rate (FDR) procedure (a rule that keeps the expected share of wrong claims among those kept below a chosen level, say 10%) decides which claims the report may state and which it should withhold.

The nearest existing tool, CiteGuard, does this for text answers to open-domain questions by checking each sentence against retrieved passages with a text-entailment model. It cannot be transplanted to serial imaging. A text model reading the prior report cannot see whether an effusion grew. Claims about one patient across many visits are correlated, which breaks the assumption that the calibration claims and the tested claims are interchangeable. And CiteGuard's own guarantee degraded by 4–6% when its Wikipedia calibration was reused in a medical domain. The contribution here is a timeline-aware version: the verifier looks at image pairs, calibration is grouped by patient and stratified by acquisition protocol, and "stable" claims are tested as seriously as "changed" ones.

Core Claim

The proposal makes two claims. First, comparative statements in serial chest X-ray reports can be held to a pre-chosen error budget $\alpha$. The guarantee is measured against radiologist-curated ground truth, not merely against the verifier. Second, three design choices make the difference between a guarantee that holds and one that silently fails: scoring with an image-pair verifier rather than a text-entailment model, calibrating at the patient level rather than the claim level, and stratifying calibration by acquisition pairing (portable AP versus standing PA). AP and PA refer to the two common X-ray directions, and they distort heart size and lung volume differently.

Concretely, for a test claim with verifier score $s$, the conformal p-value is

$$p = \frac{1 + \#\{i \le n : s_i \ge s\}}{n+1},$$

where $s_1,\dots,s_n$ are verifier scores of known-false claims in a calibration set drawn from the same stratum as the test claim. A small $p$ means the claim looks more supported than almost any false claim does. The Benjamini–Hochberg (BH) procedure, or a dependence-robust variant, is then run over all claims in a report. This p-value is valid only if a false test claim is exchangeable with the calibration false claims, meaning the two are statistically interchangeable. Much of the proposal is about earning that exchangeability when the claims come from long, correlated patient timelines.

How This Relates to Prior Work

Two sets of papers motivate the work. Longitudinal vision-language models improve how comparisons are generated but give no error bounds. DeltaNet decodes from a feature difference between the current and prior image. BioViL-T separates static from temporal features and prompts generation with the prior report. CoCa-CXR restricts attention to matching regions across time. TiBiX adds elapsed-time tokens and documents the "unchanged" trap. LUMEN extends the setting to prognosis and is judged by an LLM score. All of them evaluate with lexical overlap, label parsers, or LLM judges, and none bounds the share of false comparative claims. We do not propose a new generator. We add a filtering layer that works with any of them.

Three papers supply method. CiteGuard contributes the pipeline shape: extract atomic claims, score them, build conformal p-values from known-unsupported calibration claims, and apply BH or the Benjamini–Yekutieli (BY) procedure, a slower-to-reject variant valid under any dependence. CiteGuard switches between BH and BY heuristically when errors look correlated. We keep the pipeline and replace three of its parts: the verifier, the unit of exchangeability, and the dependence handling. Chest ImaGenome contributes the claim schema of 29 anatomical regions with improved, worsened, or no-change relations. It also contributes the labels: a silver set of more than 670,000 automatically extracted comparison relations, and a gold set of 500 radiologist-curated patients. CoCa-CXR's regional cross-attention model is the natural image-pair verifier.

Two further papers constrain the design. ADDIS shows that FDR procedures lose power when null p-values are conservative, meaning stochastically larger than uniform. That is exactly what we expect when most regions are truly unchanged and the verifier is confident about them. We therefore test whether discarding very large p-values recovers power in the offline setting. Lee et al. apply hierarchical FDR to longitudinal EHR features over a fixed ontology tree, testing broad groups before specific codes. They leave temporal structure open. We keep their top-down idea only as a secondary experiment, testing lung zones before sub-regions.

Research Program

Claim extraction. Each report sentence carrying a comparison is mapped to triples of (Chest ImaGenome region, finding, direction), with direction taking values improved, worsened, or stable. The parser is the Chest ImaGenome rule-based pipeline, which reports comparison-relation extraction F1 of 0.832. Its errors on gold reports are measured separately so they are not confused with verifier errors. A claim of stability is a hypothesis in its own right. It is false when the region actually changed. Testing stability claims is what lets the method target the "unchanged" trap directly.

Verifier. A regional image-pair change classifier is trained on the silver labels of training patients. It takes the current and prior scans, a region, and a finding, and outputs a score for each direction. The verifier never reads the prior report. As a result, an error copied forward through prior-report prompting does not raise the score of its copy. This is how we break the cross-visit "avalanche" at the source rather than correcting for it statistically afterwards.

Calibration. Calibration patients are disjoint from verifier-training patients. Within each stratum (acquisition pairing crossed with finding family), we keep at most one false claim per patient. This makes calibration units exchangeable at the patient level, the level at which test patients actually arrive. We compare this against naive claim-level calibration.

Testing. Within each report we run BH, BY, and a patient-block procedure that applies BH to report-level aggregated claims, along with ADDIS-style offline discarding. For reports later in a long timeline, the test also conditions on whether the same region's claim at the previous visit was rejected. This lets us study the multi-visit setting beyond the two-scan comparisons that every cited model is limited to.

Guarantee scope. If calibration labels come from silver data, the guarantee is relative to silver labels. We therefore report realized FDR on the gold set and on MS-CXR-T separately, and we state the gap between the two.

Baselines

The naive reference point is emitting every comparative claim the generator writes, with no filtering. Its realized FDR on gold labels is the error rate clinicians face today. There is no reported result on Lacuna for the false-claim rate of longitudinal report generators. The closest analogue is vanilla retrieval-augmented generation at 30.6% FDR on Natural Questions, reported by CiteGuard.

The strongest existing filtering method is CiteGuard transplanted as faithfully as possible. A DeBERTa entailment model scores each claim against the prior report and the current report's other sentences. It is recalibrated on 200–500 in-domain unsupported clinical claims to remove its documented domain penalty, and it uses its adaptive BH/BY switch. On Natural Questions CiteGuard reached 9.4% FDR at $\alpha=0.10$ while retaining 86–92% of supported claims. Our method beats it if it holds realized gold-set FDR at or below $\alpha$ where the transplant does not, or holds FDR equally while retaining more true claims. Both systems use the same claims, the same calibration patients, and the same calibration-set size.

The strongest image-pair model serves as the verifier baseline. CoCa-CXR reports 65.0% average accuracy on MS-CXR-T across five conditions, 4.8 points above BioViL-T. Chest ImaGenome reports 75.3% accuracy for predicting localized change. We use a CoCa-CXR-style regional model with a raw score threshold tuned on validation data at the same patients and tuning budget. This isolates what calibration adds beyond a strong verifier.

Two ablations isolate the key ingredients. The first keeps everything else fixed and replaces patient-level stratified calibration with claim-level pooled calibration. The second runs the full pipeline with the text-entailment verifier substituted for the image-pair verifier.

Why not just…?

Why not just threshold the image-pair verifier's confidence? A threshold tuned on validation data controls error only on average over that cohort. Nothing ties it to the error share among the claims that survive in any particular kind of report. It also cannot adapt when the base rate of true change shifts, as it does between intensive-care portable films and outpatient PA films. CiteGuard's 30.6% unfiltered FDR shows how badly fluent output can miscalibrate. Still, a well-tuned threshold may be close enough on in-distribution data, so we treat it as a real competitor. The decisive test is realized FDR across strata with different change prevalence: the threshold should drift while conformal BH should not.

Why not just apply CiteGuard with medical recalibration? This is the most serious threat, because recalibration removes CiteGuard's known weakness. Its remaining limitation is that a text-entailment model can only check consistency with other text. A "stable" claim copied from a wrong prior report is entailed by that prior report, and CiteGuard would keep it. We expect the transplant to keep high FDR specifically on stability claims and on patients whose prior report was wrong. The verifier-swap ablation measures exactly this. If the transplant matches us there, the image verifier is not needed and we will say so.

Why not just use Bonferroni or BY for safety? Both are valid, but dividing $\alpha$ across the dozens of region-finding claims in a report is likely to suppress most true changes, and Chetverikov et al. document how conservative such corrections become in high dimensions. We report their retention of true claims alongside BH. If BY retains nearly as much, it becomes the recommended default.

Why not just ask a large LLM judge, as LUMEN does? A judge's 1–10 score is not calibrated to any error rate. CiteGuard found that prompting-based verifiers such as Self-RAG and Chain-of-Verification both pruned valid claims and missed errors. We include a judge's scores as an alternative verifier inside our calibration pipeline. That gives it a fair chance to work and turns the comparison into a question of which scorer yields better retention at equal guaranteed FDR.

Proposed Experiments

Data come from MIMIC-CXR serial studies with Chest ImaGenome silver labels, split by patient into verifier-training, calibration, and test sets. The Chest ImaGenome gold set (500 patients) and the 1,326 labeled MS-CXR-T image pairs are reserved for final evaluation. Claims come from two sources. The first is actual MIMIC radiologist reports, which tests whether the method filters human-written comparisons sensibly. The second is a reproducible longitudinal generator, a DeltaNet-style retrieve-and-update model. Metrics are realized FDR against gold labels at $\alpha \in \{0.05, 0.10, 0.20\}$, retention of true claims (the proportion of truly correct claims that are kept), and both broken down by claim direction, acquisition pairing, and position in the timeline.

Three checks decide the proposal. The first is whether realized gold-set FDR stays at or below $\alpha$ for our method but exceeds it for claim-level pooled calibration on patients with five or more studies. That would show patient-level exchangeability matters. The second is whether the CiteGuard transplant exceeds $\alpha$ on stability claims following an erroneous prior report while our image-pair verifier does not. That would show the image-only verifier is necessary. The third is whether stratifying by AP/PA pairing removes an FDR violation that appears in AP-to-PA comparisons under unstratified calibration. That would show the method holds up under acquisition shift.

If the first two checks fail, the simpler alternatives are sufficient and the contribution narrows to a benchmark measurement of comparative-claim error rates. That result is still unreported in the literature.