Chest X-ray classifiers learn from labels mined out of radiology reports, but those labels say only what was seen, never where. The report itself usually does say where: "small left pleural effusion," "right lower lobe opacity." We propose to turn that sentence-level location into training pairs. A text-mining model extracts each finding together with its stated anatomical location. A segmentation model turns the location into a region of the image. A diffusion model, which is a generator that learns to turn random noise step by step into realistic images, then repaints only that region to add or remove the finding while leaving every other pixel alone. Each pair of images differs in exactly one clinical fact at one known place. That gives a direct signal for teaching a model to respond to the finding and not to whatever usually comes with it, and a direct test of whether its explanations are honest.
Restricting inputs to the lungs is known to remove gross shortcuts such as spine brightness and hardware. It does not remove shortcuts inside the lungs. Effusion co-occurs with atelectasis, edema co-occurs with cardiomegaly, and a classifier can score one finding by reading another. Interpretable "concept" models inherit the same problem, because their concepts are supervised with the same co-occurring labels. Paired, located edits address this within-region confounding. They also let us measure it, which no existing benchmark in this area does.
Core Claim
Our central claim is that report-grounded, region-restricted counterfactual pairs make a better training and auditing signal than three cheaper alternatives: more unpaired synthetic images, input masking, and spatial penalties on explanations. The pairs should make a concept model's per-region concepts respond selectively to the edited finding in the edited region, and this selectivity should carry over to real images with expert masks. The generator is a replaceable part. The claim concerns the pairing and the report-derived location, and the experiments are designed so that a cheaper editor could take over if it works as well.
This differs from the closest work in specific ways. Diff-CXR generates whole chest X-rays from compressed report text plus 14 CheXbert disease labels. It produces new unpaired images with no spatial control, and its compression step discards exactly the location modifiers we rely on. CoFE builds counterfactuals by pasting rectangular patches between real patients until a classifier flips. Its edits are located by the classifier, not by the report, and they leave visible seams. P-CSS breaks disease co-occurrence by masking features at coarse prototype locations, not at report-stated anatomy. Counterfactual CBMs and their follow-up compute counterfactuals only as abstract concept vectors and never touch pixels. Coherent concept explanations confine concept activations to a lesion mask with a Dice-style penalty. That constrains where a concept may fire but not what it responds to inside the region. We take that work as the main discriminative competitor.
Research Program
The first component turns reports into located findings. We run a RadGraph-style entity-and-relation model on MIMIC-CXR reports. That paper reports entity F1 0.94 and relation F1 0.82 on MIMIC-CXR. From its output we read Observation–Located At–Anatomy triples, with negation taken from the extracted modifiers. We map anatomy terms onto a coarse vocabulary of about twelve image regions: left and right lung split into upper, middle and lower zones, the costophrenic angles, the cardiac silhouette and the mediastinum. We obtain these regions geometrically from a lung segmenter of the XLSor type, which reached a mean IoU of 0.938 in the lung-segmentation study, together with a heart mask. The vocabulary is deliberately coarse. Annotator agreement on fine relations is only moderate (κ = 0.397 on CheXpert relations in RadGraph), and false precision would corrupt the edits.
The second component is the editor. We fine-tune a latent diffusion model for inpainting on MIMIC-CXR. It is conditioned on the region mask and on a small structured code, the (observation, region, present/absent) triple, rather than on free text. This conditioning choice takes a side in the tension between Diff-CXR's text compression and RadGraph-XL's fine detail: it keeps the location and drops everything else. During training, the mask covers the report-stated region of real positive images. To add a finding, we inpaint the region of a negative image with "present." To remove one, we inpaint a positive image with "absent." Pixels outside the mask are copied back exactly, so every difference between the two images lies inside the region by construction. The editor is the main risk: it may produce realistic-looking texture that is anatomically wrong, which is the known hallucination failure of these models. We therefore filter edits with an independent classifier trained only on real data, and we audit a sample with radiologists.
The third component is the model that uses the pairs. It is a concept bottleneck model, meaning one that must first predict human-readable concepts and then predict the diagnosis from those concepts alone (Concept Bottleneck Models). Its concepts are the (observation, region) pairs, computed by mask-weighted pooling of features within each region, following the mask-weighted COVID CT triage and lesion-centric breast ultrasound work. Besides the usual concept and label losses, each edit pair $(x, x')$ that adds observation $o$ in region $r$ contributes a selectivity loss:
$$ \mathcal{L}_{\text{pair}} = \max\big(0,\ m - (\hat c_{o,r}(x') - \hat c_{o,r}(x))\big) + \lambda \sum_{(o',r') \neq (o,r)} \big|\hat c_{o',r'}(x') - \hat c_{o',r'}(x)\big|. $$
Here $\hat c_{o,r}(x)$ is the model's predicted concept score for observation $o$ in region $r$ on image $x$, and $m$ is a margin. The first term requires the edited concept to rise by at least $m$. The second term, weighted by $\lambda$, requires every other concept to stay put. The second term is what attacks within-region co-occurrence: adding an effusion must not raise the atelectasis score, because nothing about atelectasis changed.
Baselines
The naive reference point is a DenseNet-style classifier trained on whole images with CheXbert labels and standard rotation, scaling and crop augmentation. CheXbert reports labeler F1 0.798. It shows what we get by doing nothing about shortcuts.
The input-masking baseline crops to the segmented lungs before classification, as in the lung-segmentation study, which reports 0.946 accuracy on COVIDx. It is the strongest cheap fix for shortcuts outside the lungs.
The spatial-penalty baseline is our concept model with region masks and the Dice-style coherence loss of Coherent concept explanations, but no pairs. That paper reports 96.00% accuracy against 85.33% for a black box on PH². It is the strongest competitor on the interpretability question, because it already enforces "concepts fire in the right place."
The unpaired-synthesis baseline adds the same number of synthetic images from global report-conditioned generation in the manner of Diff-CXR. That paper reports FID 19.50 and mAUC 0.678 on MIMIC-CXR, and states that 1% real data plus synthetic data approaches 100% real data. We found no exact downstream number for that augmentation on Lacuna. This is the strongest competitor on the data-efficiency question.
We also run two ablations. The first is a patch-swap pair baseline: located copy-paste of a real positive region into a negative image, with blended boundaries, which is CoFE's mechanism with report-derived location. The second replaces the report-stated region with a random lung region, which isolates the contribution of the report-derived location.
All arms share the backbone, the MIMIC-CXR official splits, the real-data fractions (1%, 10%, 100%), the count of added synthetic images, and a fixed hyperparameter search of 20 trials per arm. Beating a baseline means one of two things. Either we improve the real-image localization and selectivity measures below with confidence intervals that exclude zero, or we match its in-distribution AUROC within one point while improving those measures.
Why not just…?
Why not just crop to the lungs? Cropping removes the spine, the sternum and external hardware. It does nothing for a lower-zone opacity that a model scores by reading a co-occurring effusion in the same crop. It also discards context outside the crop, such as the cardiac silhouette for edema. The masking baseline above measures this directly: if it equals our method on the co-occurrence-split evaluation, the pairs add nothing.
Why not just add more synthetic images, as Diff-CXR does? Unpaired images inherit the co-occurrence statistics of the reports they are conditioned on, so they add volume without breaking confounds. Diff-CXR also does not check where its generated findings land. This baseline is a real threat on plain AUROC, and we expect it may tie there. The decisive comparison is on selectivity and the co-occurrence split, where pairing should matter.
Why not just use the coherence loss? A spatial penalty keeps a concept's activation inside its region but cannot tell two findings in the same region apart. This is the most direct test of the core claim. If the coherence-loss model matches ours on selectivity, the claim fails.
Why not just paste real patches instead of running a diffusion model? This is the cheapest serious alternative, and it could work as well. Real texture also avoids hallucination. We therefore treat it as a co-equal arm rather than a straw man. If located copy-paste matches diffusion editing on the real-image checks, we will adopt it and report that the generator is unnecessary. The claim about report-located pairing would still stand.
Proposed Experiments
The first check measures localization on real images with expert masks. We train on MIMIC-CXR and evaluate on SIIM Pneumothorax, which has pixel-level masks. The measure is the fraction of the pneumothorax concept's evidence that falls inside the expert mask, scored with the pointing game and with mass-in-mask. No arm ever trains on these masks.
The second check is a co-occurrence split. We build a real-image test subset of MIMIC-CXR in which, according to the extracted report graphs, a target finding appears without its usual companion (for example, effusion without atelectasis) or the companion appears alone. We report AUROC on the full test set and on this split. The gap between the two measures shortcut reliance. Our prediction is that pair training narrows the gap more than any baseline, and that the random-region ablation narrows it less. That second prediction is the test that report-derived location carries the effect.
The third check measures selectivity on edits that no model was trained against. Here a second editor, trained with a different seed and data split, produces the edits, so that we do not grade the model on its own generator's artifacts. We report the edited concept's rise and the mean change in the other concepts.
The fourth check is an editor audit. Two radiologists rate 200 edits, without knowing which arm produced them, for plausibility and for whether the finding sits in the stated region. We also report how often the independent real-data classifier detects the intended finding in each edit.
We also test external transfer to IU X-Ray. The project fails clearly if the pair-trained model does not beat the coherence-loss and copy-paste arms on the SIIM localization measure and on the co-occurrence-split gap. It also fails if radiologists judge more than a small minority of edits misplaced. In that case the report-derived locations, and not the model, are the weak link.
References
- Diff-CXR: Report-to-CXR generation through a disease-knowledge enhanced diffusion model
- Contrastive Learning with Counterfactual Explanations for Radiology Report Generation
- Rethinking Radiology Report Generation via Causal Inspired Counterfactual Augmentation
- Counterfactual Concept Bottleneck Models
- Climbing the Ladder of Interpretability with Counterfactual Concept Bottleneck Models
- Coherent Concept-based Explanations in Medical Image and Its Application to Skin Lesion Diagnosis
- RadGraph: Extracting Clinical Entities and Relations from Radiology Reports
- Improving Classification Model Performance on Chest X-Rays through Lung Segmentation
- Concept Bottleneck Models
- Computed tomography–based COVID–19 triage through a deep neural network using mask–weighted global average pooling
- Lesion-Centric Latent Phenotypes from Segmentation Encoders for Breast Ultrasound Interpretability
- Combining Automatic Labelers and Expert Annotations for Accurate Radiology Report Labeling Using BERT