Learning Where a Chest X-ray Model Will Break, from the Reports Radiologists Already Wrote

Image classifiers for chest X-rays work well at the hospital that trained them and fail elsewhere. A DenseNet trained on NIH ChestX-ray14 falls from 0.83 to 0.68 mean AUC on Stanford's CheXpert (Can We Trust Deep Learning Based Diagnosis?). Part of the reason is that models pick up shortcuts. A chest tube "means" pneumothorax only because pneumothorax gets treated with a chest tube, and how often patients are imaged after treatment differs from hospital to hospital. The information needed to see these shortcuts is already written down. Radiology reports routinely say "portable AP view," "chest tube in place," or "monitoring leads overlie the chest." The proposal is to turn those sentences into variables and then learn a causal graph across several hospitals. The graph is a diagram of what causes what, and it includes an explicit "which hospital" node. From that graph we read off which shortcut paths depend on the hospital and use that to decide what the image model should be made insensitive to.

The second goal matters as much as the first. The graph should predict, before deployment, which patient subgroups will collapse at a new hospital. The plain-language version is short. An LLM reads the reports and pulls out the non-disease facts in them. A causal discovery method then works out which of those facts are tied to the disease differently at different hospitals, and those are the facts the image model must not rely on.

Core Claim

Hospital-specific shortcut paths can be discovered automatically by combining two things. The first is evidence-grounded extraction of acquisition and treatment factors from free-text reports. The second is a "hybrid" causal discovery method, meaning a statistical search over graphs that also uses an LLM's world knowledge as a soft prior on edge directions. The hospital is placed in the graph as a variable with no causes. The output is an automatically learned version of the hand-drawn "augmented causal graphs" of A Causal Framework for Mitigating Data Shifts in Healthcare. That paper adds explicit shift variables to a causal diagram to show where environments intervene, but every diagram in it is built by expert deliberation.

We claim the learned graph is useful in two ways that a flat list of extracted attributes cannot be. It separates factors whose link to the label is unstable across sites from factors that carry genuine, stable signal. It also orients edges such as pneumothorax → chest tube, so that a device is recognized as a downstream effect of treatment and not as a cause. Deconfounding only the graph-selected factors should match or beat deconfounding everything we extracted on worst-group accuracy, while losing less average AUC. The graph learned on training sites should also rank which subgroups will degrade at a held-out site.

Research Program

The work has three stages: extract factors from reports, learn a graph across sites, and use the graph to train and to forecast.

Stage 1: extraction. We build a per-study table of candidate non-pathology factors from MIMIC-CXR reports. These cover view and position, portable versus departmental acquisition, each support-device type, recent surgery, and follow-up status. The pipeline follows the extract, find-omissions, cite-the-evidence, prune sequence of Self-Verification Improves Few-Shot Clinical Information Extraction. That pipeline raised MIMIC-IV ICD-10 F1 from 0.487 to 0.533, and its cited text spans overlapped human annotations 93% of the time. Every factor value therefore comes with a verbatim span a clinician can audit.

Pathology labels come from CheXbert (Combining Automatic Labelers and Expert Annotations for Accurate Radiology Report Labeling Using BERT), which reports 0.798 average F1. For CheXpert we use the released labeler outputs, which already include support devices; SpurCon uses this column as its spurious cue. We may also add text-derived factors where report text is accessible to us. This step is not optional bookkeeping. Automated labelers carry a 10–30% error rate, and CheXpert and MIMIC-CXR share labeler logic (Can We Trust Deep Learning Based Diagnosis?). To keep labeler differences from masquerading as clinical shift, we add a "label source" indicator to the graph. We also run the same extractor on every site that has text, and audit about 500 reports per site against radiologist reading.

Stage 2: cross-site graph learning. Following the propose-and-verify loop of COAT (Discovery of the Hidden World with Large Language Models), an LLM proposes candidate factors from report samples. When the statistical model fits a cluster of studies poorly, those studies are fed back so the LLM can propose factors it missed. COAT runs on single homogeneous datasets and has no notion of site. We change the discovery step itself.

The pooled multi-site table gets a site node $S$. We forbid any edge into $S$, and we search over graphs $G$ with the penalized score of Regularized Multi-LLMs Collaboration for Enhanced Score-based Causal Discovery:

$$\hat G = \arg\max_G \; \text{Score}_{\text{data}}(G) - \lambda \cdot \text{Penalty}_{\text{LLM}}(G),$$

Here $\text{Score}_{\text{data}}$ is a decomposable fit score such as BIC on the pooled table. $\text{Penalty}_{\text{LLM}}$ counts disagreements with an ensemble of LLM judgments about edge direction, and $\lambda$ sets how much the LLM prior is trusted. That paper leaves the choice of $\lambda$ as a manual heuristic. We instead choose it by leave-one-site-out held-out likelihood. The prior is trusted exactly as much as it helps the graph transfer to a site it has not seen.

From $\hat G$ we read off three kinds of path. A factor $F$ is hospital-unstable for label $Y$ when it lies on a path $S \to F$ that is also linked to $Y$. A factor is a stable predictor when its dependence on $Y$ is invariant given $S$. A factor is a downstream effect of $Y$ when the edge runs $Y \to F$, as with a treatment device.

Stage 3: using the graph. The unstable factors define groups for a deconfounding objective. We reuse SpurCon's weighted contrastive loss (SpurCon) but replace its two hand-supplied inputs: its human-chosen cue becomes our graph-selected factors, and its 10-shot visual prototypes become our text-derived group labels. Separately, we turn the graph into a forecast. For a held-out hospital we compute which factor–label associations the graph implies will change, given that site's factor marginals as estimated from its reports. We then predict which subgroups' accuracy will fall.

The approach has clear limits. It inherits the faithfulness and no-hidden-confounder assumptions that COAT also carries. We test sensitivity by running a latent-confounder-tolerant search (FCI, as COAT-PAG does) alongside the score-based one. It also cannot see shifts that reports never mention, such as scanner brand. Those remain the territory of pixel-level methods.

Baselines

The naive reference is a DenseNet121 trained by ordinary empirical risk minimization with standard augmentation on the pooled training sites. This is the model whose cross-site drop from 0.83 to 0.68 AUC is reported in Can We Trust Deep Learning Based Diagnosis?. We report both its average AUC and its worst-group accuracy on held-out sites. Every method shares the same backbone, the same frozen-encoder-plus-head option where applicable, the same training sites, and the same hyperparameter budget. The budget is a fixed number of trials, with selection on a validation split drawn from training sites only. The held-out site is never used for tuning.

The strongest existing method on the shortcut question is SpurCon with its human-specified support-device cue. On CheXpert pneumothorax it reports worst-group accuracy of 73.0%, up from 35.6% for the baseline (SpurCon). We rerun it on our splits rather than copying the number, because our splits are multi-site. Our method uses the same loss, so any difference comes from which cues are chosen and how they are labeled.

For representation robustness, the strongest competitor is report-image contrastive pretraining (CLIP-style) on MIMIC-CXR. It reaches 78.19 AUROC on CheXpert with 1% labels, against 72.83 for SimCLR (Radiology Reports Improve Visual Representations Learned from Radiographs). We start our method from this same pretrained encoder, so the comparison asks only whether graph-guided deconfounding adds anything on top of it.

Three ablations isolate the key ingredient. The first, deconfound-everything, applies the same loss to all extracted non-pathology factors with no graph. The second, zero-shot-LLM-selected, asks the LLM directly which factors are spurious, with no data score. The third, data-only, sets $\lambda = 0$ and uses no LLM prior. Beating the field means two things. On worst-group accuracy we must beat SpurCon-with-known-cue on CheXpert pneumothorax, or match it while needing no human cue specification. On a held-out site we must exceed deconfound-everything on worst-group accuracy while staying within one AUC point of the best average AUC.

Why not just…?

Why not just deconfound every non-pathology attribute the extractor finds? This is the real threat. If it works as well, the causal graph is ornamental. Our hypothesis is that it over-corrects. Some extracted factors carry stable diagnostic signal, and removing them throws away genuine signal; A Causal Framework for Mitigating Data Shifts in Healthcare warns of exactly this for entangled clinical variables. Grouping on many attributes also fragments the data into tiny groups. The deconfound-everything ablation settles the question directly. If it ties our method on both worst-group accuracy and average AUC, we will report that the graph's value lies only in the forecasting use, and we will narrow the claim accordingly.

Why not just ask GPT-4 which factors are spurious? LLMs are strong at pairwise direction questions; GPT-4 reaches 97% on Tübingen pairs (Causal Reasoning and Large Language Models). Whether a factor is spurious at these particular hospitals, however, depends on local treatment and imaging policy, and no world knowledge contains that. On multi-variable structure, zero-shot LLM discovery scored 0.46 Markov-blanket F1 against COAT's 0.88 on AppleGastronome (Discovery of the Hidden World with Large Language Models). The zero-shot ablation tests whether the same gap appears here.

Why not just use CLIP on the reports? Contrastive pretraining pulls the image representation toward everything the report says, including "chest tube in place." It therefore has no mechanism to discard a shortcut that is spelled out in text. We keep CLIP as our starting encoder. If adding graph-guided deconfounding does not raise its held-out worst-group accuracy, the proposal fails on that axis.

Why not just run JTT or group reweighting with no extracted factors? SpurCon already reports that it outperforms JTT on CheXpert and trains 5.5–9× faster (SpurCon). Reweighting by error also cannot say which subgroup will fail at an unseen hospital, so it offers nothing for the forecasting claim. We include JTT for completeness.

Proposed Experiments and Decisive Checks

The main study rotates leave-one-hospital-out across MIMIC-CXR, CheXpert, and PadChest. ChestX-ray14 serves as an image-only target with no report-derived factors. Targets are pneumothorax, edema, and pleural effusion, and metrics are average AUC and worst-group accuracy over graph-defined and device-defined groups. We also test the method outside chest X-rays on ISIC 2020, where SpurCon lifts ruler-artifact worst-group accuracy from 0% to 65.0%. That test uses metadata and any lesion descriptions available, and we will report plainly if text coverage there is too thin.

Three checks are decisive. The first tests graph recovery when the true answer is known. We implant a synthetic marker that correlates with the label at one training site only and mention it in that site's reports. The marker's correlation strength and its direction relative to the label vary across conditions. The graph must place the marker on an $S \to F$ path, and we measure structural Hamming distance against the implanted structure for the hybrid, data-only, and LLM-only variants. For context, the hybrid score cut SHD from 4 to 2 on Earthquake (Regularized Multi-LLMs Collaboration). No reported result was found on Lacuna for any graph-recovery metric on chest X-ray factor tables.

The second check tests forecasting. The Spearman correlation between the graph's predicted per-subgroup degradation and the observed drop at the held-out site must clearly beat two simple forecasts. One uses factor-prevalence differences between sites; the other uses the deconfound-everything model's training-site subgroup gaps.

The third is the over-deconfounding test described under the first alternative. Graph-selected deconfounding must keep at least one AUC point more than deconfound-everything at equal or better worst-group accuracy. If none of the three checks favors the graph, we conclude that report-derived attributes alone are what matters, which would be a useful negative result for anyone tempted to add causal machinery here.