When a model writes a chest X-ray report for a patient who has been imaged before, much of what it says concerns change: "effusion has increased," "new right basilar opacity," "pneumothorax resolved," "unchanged since the prior." Current automatic metrics for these reports compare one generated report with one reference report written for the same exam. They cannot tell whether a claim about change is actually true of the patient's history. They inherit that blindness even when they are carefully designed for clinical content. We propose a metric that reconstructs each finding's history across all of a patient's prior reports. It links every mention of "the left effusion" across visits into a single thread, then checks each change claim in a generated report against the thread it belongs to.
The linking step is where the second research direction comes in. It is a multiple-sequence alignment problem: with $d$ visits and $m$ findings per report, there are on the order of $m^d$ candidate threads. That exponential growth is the same one that multiple-sequence alignment methods in bioinformatics are built to tame. We borrow their main trick. A cheap pairwise calculation gives an optimistic bound on the full multi-visit score, and search guided by that bound finds the best threads exactly while exploring only a small fraction of the space. The two-sentence version is this: longitudinal report evaluation needs to know which findings across a patient's reports are the same finding, and that linking can be solved exactly and cheaply with pairwise-bounded search. Whether the search actually beats simpler linking rules is the central empirical question.
Why current metrics miss this
The clinical metrics built so far all score a single pair of reports. RadGraph turns a report into a graph: anatomy and observation nodes, each observation marked present, uncertain, or absent, joined by "located at," "suggestive of," and "modify" relations. Evaluating Progress in Automatic Chest X-Ray Radiology Report Generation scores overlap of these graphs (RadGraph F1) and combines it with BLEU into RadCliQ. RaTEScore matches medical entities through synonym-aware embeddings but treats a report as an unordered set of entities. GREEN fine-tunes a language model to count clinically significant errors between a candidate and a reference. We reuse the first three as components: the graph schema, the extractor, and the entity similarity. We do not reuse their single-pair scope.
The longitudinal generation literature supplies the motivating evidence. A review of longitudinal report generation documents that models hallucinate comparisons, yet every benchmark it surveys still scores with single-report metrics. DDaTR and the longitudinal pre-fill model condition on exactly one prior study. Both report weakness when a finding flips state between visits, and both evaluate with BLEU, ROUGE and label-based clinical efficacy.
The gap is concrete. A reference report often says "moderate left effusion" without any comparative language. A generated report that says "left effusion has increased" can then match the reference's entities perfectly while making a claim that is false if the effusion was large two studies earlier. Reports also routinely omit stable findings. As a result, the relevant prior mention may lie several visits back, beyond the single-prior window these models use.
The closest methodological relative is ASSESS. It builds a domain metric for formal math statements by searching over tree edits guided by rewrite rules. It shows that structured search can beat string metrics for expert agreement (Cohen's kappa 0.35 versus 0.26 for BLEU on EPLA-miniF2F). However, it compares only two statements and works on exact formal syntax. Our search runs over many visits and over fuzzy, uncertain clinical assertions.
Core Claim
Our claim is that change claims in generated reports should be verified against finding threads reconstructed from the full reference history. We claim these threads can be found by exact bounded search over a multi-visit alignment. That search's objective includes transition-consistency terms that pairwise linking cannot express, and the claim is only worth making if those terms matter for agreement with radiologists. We state it this way because the simplest version of the idea needs no clever search at all, as the next section explains.
The alignment problem and where it becomes hard
Each reference report $R_t$ for visits $t=1,\dots,d$, together with the generated report, is parsed into RadGraph-style findings. Each finding carries an observation, an anatomy, a presence state, and modifiers such as size. A thread $\tau$ selects at most one finding per visit, or a "not mentioned" symbol $\perp$. A set of threads $T$ is scored as
$$S(T)=\sum_{\tau\in T}\Big[\sum_{t} a(\tau_t,\tau_{t'}) + \sum_{t} g(\tau_{t''},\tau_{t'},\tau_t)\Big],$$
where $t'$ and $t''$ are the most recent earlier visits at which the thread was mentioned. The term $a$ is a pairwise affinity. It combines synonym-aware entity embedding similarity (as in RaTEScore), anatomy compatibility from the "located at" relation, and a small cost for skipped visits. The term $g \le 0$ is a higher-order penalty over three consecutive mentions of a thread. It penalizes implausible state sequences, for example "resolved" followed by "unchanged," or size modifiers that contradict each other. It also treats $\perp$ as "carried forward, unknown" rather than "absent."
If $g$ were zero, the problem would split into links between visit pairs. It could then be solved exactly as a minimum-cost flow over a directed acyclic graph with skip edges, and there would be no curse of dimensionality to speak of. We say this plainly because it defines our strongest ablation. The higher-order terms, together with many-to-one grouping (several granular observations summarized by one diagnosis through "suggestive of" edges), are what make the joint problem genuinely combinatorial.
The method comes from MLCS-A*. That work bounds the common subsequence of $d$ strings by the minimum over pairwise common subsequences. The bound never underestimates what remains, so A search stays exact while pruning heavily, and the authors report over 800× speedups from the approximate variant. In our setting, $g \le 0$ means that the flow solution on the remaining visits, with $g$ dropped, is an optimistic bound on any completion. It is therefore an admissible heuristic for A over partial thread assignments. From mini-MLCS we take the practice of pruning a partial assignment as soon as its bound falls below the best complete solution found so far, rather than building the full graph first.
The research file flags one constraint as an open problem: such pruning works when matches are sparse and degrades on dense, soft similarities. We therefore threshold $a$ to keep the candidate links sparse and measure how optimality and runtime change with that threshold.
From threads to a score
Once threads are built over the reference history plus the generated report, each change claim in the generated report is identified. These are findings carrying modifiers such as new, increased, decreased, resolved, unchanged, or an explicit comparison date. Each claim attaches to a thread and is labeled supported, contradicted, or unverifiable. The label depends on the thread's states at the claimed comparison point and at the current visit, where the current state comes from the reference report.
The metric reports two numbers. Temporal claim precision is the share of the generated report's change claims that the reference threads support. Change-event recall is the share of reference-thread state changes at visit $t$ that the generated report mentions. These sit alongside any existing single-report score rather than replacing it.
The "unverifiable" label is deliberate. CareTransition-Audit found that model auditors force yes/no answers where clinicians say "unclear." A metric that collapses unknowns into errors would repeat that failure.
Research Program
The first step is a measurement that could make the rest unnecessary. We rebuild patient histories from MIMIC-CXR study dates and the patient-level linkage used by Longitudinal-MIMIC and the ULCX subset. ULCX has 95,168 samples from over 26,000 patients, so many patients have more than two studies. Among patients with at least four studies, we measure how often correctly verifying a change claim in the reference report requires a visit earlier than $t-1$. If that fraction is tiny, a single-prior metric suffices. The proposal then shrinks to a two-visit claim checker, and we would report exactly that.
The second step is an annotated evaluation set. We sample patients with four or more studies and generate current-visit reports from three sources: a longitudinal model in the style of DDaTR, a single-image model, and a language model asked to rewrite reference reports with controlled edits. Radiologists label each change claim as supported, contradicted, or unverifiable given the prior images and reports. Two radiologists label an overlapping subset so we can report a human agreement ceiling. That ceiling matters: RadGraph's own relation agreement was only kappa 0.397. We also build synthetic tests in the style of RaTEScore's synonym-versus-antonym probe: flipping change words, substituting a different patient's history, and deleting the only earlier mention of a finding.
The third step is the search itself. We compare exact A* with the flow-based bound against beam search, and record nodes expanded, runtime per patient, and how often beam search misses the optimum as $d$ grows to the longest histories available.
Baselines
The naive reference point scores the generated report against the current reference report alone, using RadGraph F1 and RaTEScore with their released extractors. This is what the longitudinal generation papers do today. It shows how much temporal error is invisible without history. On ReXVal these single-report metrics reach Kendall tau 0.515 for RadGraph F1 and 0.527 for RaTEScore (RaTEScore). The strongest single-pair method is GREEN at tau 0.63–0.64 against radiologist error counts on ReXVal (GREEN). We run GREEN's released model unchanged on (generated, current reference) pairs.
No reported result was found on Lacuna for any metric's agreement with radiologists on verifying longitudinal change claims. The comparison on our annotated set is therefore new for every method, and we report all of them under identical conditions.
The strongest competitor on this specific question is a language-model judge that sees the entire reference history. We use the GREEN model and a larger open instruction-tuned model, prompted with all prior reports plus the current reference and asked to label each change claim with the same three labels. We give its prompt design the same development split and the same number of tuning iterations that we spend setting our own weights and thresholds.
Two ablations isolate the key ingredient. The first links findings by exact match on a normalized key (observation plus anatomy) and never searches. The second drops the higher-order term $g$ and links consecutive mentions by minimum-cost flow. A third ablation truncates history to the single prior visit. All systems share the same entity extractor, the same held-out test patients, and CPU-only execution for the non-neural methods.
We count the proposal as beating a baseline if it achieves higher Cohen's kappa with radiologist claim labels, with a paired bootstrap interval excluding zero. Against the history-aware judge, matching its kappa while running deterministically on a CPU would be a partial win. Clearly exceeding it would be a full one.
Why not just…?
Why not just score against the current reference, since it already contains the radiologist's change language? When the reference states the change explicitly, pairwise metrics can catch a mismatch. But references frequently describe the current state without comparison, and a generated comparative claim then goes unchecked. The first-step measurement quantifies how often this happens. The naive baseline's kappa on contradicted claims shows directly what it misses.
Why not just give a strong language model the whole history and ask it? This is the real threat. The evidence against it is indirect. Zero-shot GPT-4 matched radiologist preferences only 23% of the time in GREEN, and model auditors were near random on the history-and-exams section of CareTransition-Audit. Neither result tested long radiology histories, though. We run this judge as a full baseline with matched tuning effort. If it wins on kappa, the honest contribution narrows to a cheap, deterministic, inspectable metric usable inside training loops, where GREEN's roughly one second per pair on an A100 is already costly. We would say so.
Why not just normalize findings to canonical keys and group by key, or chain pairwise matches between consecutive visits? Either could plausibly be enough, and it is exactly what our ablations test. Grouping by key fails when the same process is renamed over time, for example opacity becoming consolidation becoming pneumonia. Pairwise chaining cannot penalize state sequences that are locally plausible but globally contradictory. Whether these cases are common enough to move kappa is an empirical question. If the flow ablation ties the full method, we drop the higher-order search and ship the simpler metric.
Why not just use the single prior study, as current longitudinal models do? Because reports often omit stable findings, the relevant earlier mention may lie several visits back. The history-truncation ablation and the first-step measurement settle this directly.
Proposed Experiments
The decisive checks are three, run on the annotated set and the synthetic probes.
The first compares kappa with radiologists on change-claim labels across the full method, the flow ablation, the key-grouping ablation, the single-prior ablation, and the history-aware language-model judge. The full method must beat the flow ablation for the multi-visit search to justify itself.
The second is a set of synthetic probes. Swapping in another patient's history should drive temporal claim precision down while leaving single-report metrics unchanged. Flipping "increased" to "decreased" should be detected at a rate we report next to RaTEScore's 67% synonym-versus-antonym accuracy as a reference point.
The third is a search study. Exact A* should expand far fewer nodes than exhaustive enumeration, and beam search's rate of missing the optimum should rise with history length. That rise would show exactness matters for the longest, sickest patients, who are the ones where change claims carry the most clinical weight.
As a sanity check, on ReXVal, which contains no history, the metric's current-visit component should not fall below RadGraph F1's tau of 0.515, since it is built from the same graphs.
References
- RadGraph: Extracting Clinical Entities and Relations from Radiology Reports
- Evaluating progress in automatic chest X-ray radiology report generation
- RaTEScore: A Metric for Radiology Report Generation
- GREEN: Generative Radiology Report Evaluation and Error Notation
- A Review of Longitudinal Radiology Report Generation: Dataset Composition, Methods, and Performance Evaluation
- DDaTR: Dynamic Difference-Aware Temporal Residual Network for Longitudinal Radiology Report Generation
- Utilizing Longitudinal Chest X-Rays and Reports to Pre-fill Radiology Reports
- ASSESS: A Semantic and Structural Evaluation Framework for Statement Similarity
- A Fast Heuristic Search Algorithm for Finding the Longest Common Subsequence of Multiple Strings
- A fast and efficient path elimination algorithm for large-scale multiple common longest sequence problems
- CareTransition-Audit: A Benchmark to Audit Discharge Summaries for Efficient Care Transitions