Healthy-Patient Baselines for Chest X-ray Attribution

Gradient attribution methods explain a classifier by asking how its output changes as an image moves from a "neutral" reference image to the real one. On chest X-rays the default reference is an all-black image. That image is not neutral. It is a picture of total photon absorption that no scanner ever produces, so the model's behavior along the way to it reflects whatever the network does on inputs it has never seen. A radiologist's question is different: what about this film differs from a healthy chest like it? This note proposes to answer that question directly. The reference becomes a small set of real, pathology-free radiographs retrieved for each patient and spatially aligned to them. We then measure how much of the documented gap between saliency heatmaps and radiologist annotations closes when the reference is changed and nothing else is.

The opportunity matters because the strongest evidence against post-hoc heatmaps in radiology was gathered entirely with arbitrary references. On CheXlocalize, Benchmarking saliency methods for chest X-ray interpretation found saliency 24.0% worse than radiologists in overlap and 29.4% worse in hit rate. Assessing the (Un)Trustworthiness of Saliency Maps found five of eight methods unchanged when model weights were randomized. Nobody has separated how much of this failure belongs to gradients themselves and how much belongs to the reference. A second payoff follows from the same machinery. Once there are several plausible healthy references per patient, the disagreement among their heatmaps is a per-image warning that a map should not be trusted. That is the kind of signal needed against the automation bias documented by Gaube et al., where 27.54% of radiologists followed incorrect advice every time.

Core Claim

We claim two things. First, for path-based attribution on chest X-rays, replacing the black reference with a distribution of retrieved, registered, pathology-free radiographs closes a substantial part of the localization gap on CheXlocalize. We expect most of that gain to come from two ingredients. One is that the references are healthy, which gives a one-versus-one contrast. The other is that they are anatomically aligned. Whether the reference was generated rather than retrieved should matter little. Second, the spread of attributions across those references predicts which individual heatmaps miss the radiologist's annotation. That spread can serve as a reliability flag for selective display.

Concretely, Integrated Gradients (IG) attributes to pixel $i$ the quantity $(x_i - x'_i)$ times the average gradient along the straight line from reference $x'$ to input $x$ (Axiomatic Attribution for Deep Networks). We replace the single $x'$ with an expectation over references drawn from a patient-conditioned healthy distribution:

$$\phi_i(x) = \mathbb{E}_{x' \sim H(x)}\left[(x_i - x'_i)\int_0^1 \frac{\partial F(x' + \alpha(x - x'))}{\partial x_i}\,d\alpha\right],$$

Here $F$ is the classifier's logit for the target pathology. $H(x)$ is the empirical distribution over the $K$ nearest "No Finding" radiographs to $x$ in the classifier's own embedding space, each affinely registered to $x$ using lung-field masks. The per-pixel reliability score is the standard deviation of the bracketed term across the $K$ references, normalized by $|\phi_i|$. The image-level flag is that score aggregated over the top-attributed region.

Relation to Prior Work

Several papers supply the method. Expected Gradients and its Grad-CAM variant, Expected Grad-CAM, already average over a reference distribution. On natural images they report infidelity of 4.7 versus 1592.0 for Grad-CAM. Their reference distribution is unfiltered training data, however, and they were never evaluated on medical images. Our change is to condition that distribution on health and on the patient's anatomy. Path-sampled integrated gradients and Shapley Integrated Gradients address variance and coalition weighting over baselines. We reuse the PS-IG deterministic quadrature to reduce cost but do not depend on its theory. Neither paper was tested on clinical localization.

Other work motivates the proposal. GANMEX showed that class-counterfactual, in-distribution references sharply improve localization. On the BAM benchmark its inverse localization score was 0.747 versus 1.591 for a zero baseline, but it was tested only on toy and natural images. It also requires training a generator, which on radiographs carries hallucination risk. The Out-of-Distribution Problem in Explainability argues that explanations computed on off-distribution inputs reflect model priors rather than learned features. Retrieval keeps the endpoint on the data distribution by construction. The straight-line path between two real radiographs still passes through blended images, and we treat that as a measured residual rather than a solved problem. Patchhealer transplants healthy segments to flip cardiomegaly decisions, and CIT-LR uses diffusion counterfactuals for fairness audits. Both show that healthy counterfactual images are feasible on chest X-rays. Neither uses them as attribution references.

Two results constrain the design. Arun et al. recommend abandoning saliency for segmentation networks. MedicalPatchNet shows an explainable-by-design patch model beating Grad-CAM in hit rate, 0.485 versus 0.376. It does so at the cost of global context for multi-region findings such as cardiomegaly with edema. Our proposal keeps a full-image classifier precisely so that such findings remain modelable. We therefore have to match or beat these alternatives on their own benchmarks rather than argue around them.

Research Program

The work has three parts. The first is a controlled "reference ladder" on a fixed set of classifiers. We will use DenseNet121 and EfficientNetV2-S trained on CheXpert, matching the backbones used by Saporta et al. and MedicalPatchNet. Each attribution method is run with a sequence of references that changes one property at a time. The sequence runs from a black image, to a blurred copy of the input, to the dataset mean, to unfiltered random training images, to random healthy images, to retrieved healthy neighbors, and finally to retrieved and registered healthy neighbors. The ladder is applied to IG and to Expected Grad-CAM, the latter with its reference set swapped. This yields a decomposition of localization error into the parts removed by each property.

The second part compares retrieval against generation at the top of the ladder. We replace retrieved neighbors with healthy counterfactuals of the same patient. These are produced either by a diffusion model in the style of CIT-LR or by Patchhealer-style segment transplants. If generation adds little over registered retrieval, the cheaper and hallucination-free option wins.

The third part validates the reliability flag. For each CheXlocalize test image we compute the across-reference spread and test whether it predicts per-image miss. A miss is a pointing failure, or overlap below a threshold, against radiologist masks. If it does, we evaluate selective display, meaning heatmaps are shown only for confidently explained images.

Retrieval costs one embedding lookup. Attribution costs $K \times m$ forward-backward passes, where $m$ is the number of quadrature steps. We will report wall-clock time against Grad-CAM's reported 0.006 s and Expected Grad-CAM's 0.115 s. We will not claim real-time suitability unless measured.

Baselines

The naive reference point is IG with a black baseline, as used by Saporta et al. Every gain on the ladder is measured from it. The fast incumbent is standard Grad-CAM, which Saporta et al. found to be the best-performing post-hoc method. MedicalPatchNet reports a CheXlocalize hit rate of 0.376 for it. We re-run it on the same classifiers rather than borrowing the number. The file contains no absolute CheXlocalize mIoU for any method, only relative gaps to radiologists, so mIoU comparisons will come from our own runs against the radiologist benchmark reported in CheXlocalize.

The strongest attribution competitor is Expected Grad-CAM with its default unfiltered reference distribution. It is the published method closest to ours and differs only in what the references are. It will receive the same number of references, the same quadrature steps, and the same smoothing-kernel tuning budget on the CheXlocalize validation split.

The strongest localization competitor on CheXlocalize is MedicalPatchNet, with a reported hit rate of 0.485. Beating it means exceeding that hit rate with a full-image classifier whose CheXpert AUROC is not lower than MedicalPatchNet's reported 0.907. We will retrain it under our data splits to confirm the number. On SIIM-ACR pneumothorax and RSNA pneumonia, the dedicated-localizer ceiling comes from Arun et al.: U-Net AUPRC 0.404 and RetinaNet AUPRC 0.596. The best saliency results there were 0.224 (XRAI) and 0.519. Our target is to beat the saliency numbers clearly and to report honestly how much of the gap to the supervised localizers remains.

Three ablations isolate the key ingredients. Random healthy versus unfiltered references isolates health. Registered versus unregistered neighbors isolates alignment. Retrieved versus generated healthy counterfactuals isolates whether the reference must be patient-specific.

Why not just…?

Why not just train a segmentation network or detector? Where dense masks exist, Arun et al. show that U-Net and RetinaNet localize better than any saliency method, and we expect that to remain true. The proposal does not try to replace them for localization. It targets the many image-level labels for which no masks exist at scale, and the task of auditing what a deployed classifier actually used. A segmentation network cannot show that, because it is a different model. Our SIIM and RSNA runs will report the remaining gap to these localizers explicitly.

Why not just use an explainable-by-design patch model? MedicalPatchNet is a real threat on hit rate. If retrieved healthy references cannot exceed its 0.485, the case for full-image attribution on focal findings weakens. Its own authors document, however, that averaging independent patch logits cannot represent findings that depend on several regions at once. We will report results per pathology so the comparison is visible separately for focal findings (lung lesion, pneumothorax) and global ones (cardiomegaly, edema, enlarged cardiomediastinum).

Why not just use Expected Gradients over ordinary training images? This is the most serious alternative because it is nearly free. If it matches the healthy, registered version, our contribution reduces to "use Expected Grad-CAM on X-rays." The ladder settles this directly. The rung of unfiltered references against healthy registered neighbors, at equal reference count and compute, is the decisive comparison. We would report a null result here as a finding, not bury it.

Why not just use a generative counterfactual like GANMEX or a diffusion model? Generation gives a patient-specific healthy image, but it may invent anatomy. Its cost and verification burden are high at radiograph resolution. The retrieved-versus-generated arm tests whether that extra specificity buys localization beyond what registration already provides. Patchhealer's domain checker, reported at 90% or higher accuracy on artifacts, will be used to screen generated references before they are compared.

Decisive Checks

The proposal succeeds if four conditions hold together on CheXlocalize test images. First, healthy registered references improve hit rate and mIoU over both the black baseline and unfiltered Expected Grad-CAM, with paired bootstrap confidence intervals excluding zero. Second, the resulting maps change substantially under cascading weight randomization, which five of eight methods failed in Arun et al. Third, across-reference spread predicts per-image misses with an area under the ROC curve well above chance; showing only the lowest-spread half of maps should raise their hit rate. Fourth, the gain persists across both backbones and over pneumothorax and pneumonia on SIIM and RSNA.

The proposal fails informatively in two cases. If unfiltered references perform as well as healthy ones, the reference's health does not matter. If no reference closes the gap, the CheXlocalize deficit is a ceiling of gradient attribution rather than an artifact of the reference, which would strengthen Arun et al.'s recommendation with a controlled experiment.

A follow-up reader study, conditional on the reliability flag working, would rerun the Gaube et al. design with heatmaps and flags shown alongside incorrect advice. It would test whether flagged heatmaps lower the rate at which clinicians follow wrong recommendations.