Does a Heatmap's Agreement with Pathology Come from the Model?

Medical imaging papers often validate a neural network's heatmap by checking whether it lands on tissue that a pathologist later confirms as cancer. A heatmap here is an attribution map: a per-pixel score meant to show which parts of the input drove the prediction. This check is weaker than it looks. A heatmap can overlap a tumor for reasons that have nothing to do with what the network learned. Prostate tumors are dark on diffusion MRI and have visible margins, and several popular attribution methods behave like edge detectors that produce nearly the same map whether the network is trained or randomly initialized. Agreement with pathology and dependence on the network's weights are two separate properties. The literature tests them separately. Heatmaps are checked against tissue in one place and checked for weight dependence in another, but no one has asked how much of the tissue agreement disappears when the learned weights are destroyed.

We propose to measure exactly that. Using prostate MRI paired with whole-mount surgical pathology, we compute each attribution method's agreement with pathologist-confirmed cancer twice: once on the trained network and once on copies of that network whose weights have been progressively randomized. The difference is the part of the biological agreement that the learned parameters actually produce. We then test this quantity where it matters most clinically: in cancer that radiologists missed but pathology found. Explanations that only copy input structure and explanations that only echo the radiologist-drawn training labels should fail differently in that region.

Core Claim

Our claim is that histological agreement only counts as evidence for an explanation when it survives a parameter-randomization control. Once that control is applied, the ranking of attribution methods on clinical data will differ from the ranking given by raw overlap with pathology. Concretely, for an attribution method $E$, a trained network with weights $\theta$, and registered pathology $H$, we define the learned histological agreement

$$\Delta_H(E) = \mathrm{AUC}_H(E;\theta) - \mathbb{E}_{\tilde\theta}\left[\mathrm{AUC}_H(E;\tilde\theta)\right],$$

The terms are defined as follows. $\mathrm{AUC}_H$ is the area under the ROC curve for using attribution values to separate cancer voxels from benign voxels, with both sets confined to the prostate gland so that the gland boundary cannot inflate the score. $\tilde\theta$ ranges over weights randomized layer by layer, both top-down and bottom-up, across several seeds.

The rank-1 convergence theory gives a sharp prediction (When Explanations Lie: Why Many Modified BP Attributions Fail). That work proved that backward passes which discard negative relevance collapse onto a fixed, input-determined pattern. It reported cosine similarity convergence above 0.99 for Guided Backpropagation, the non-negative z+ rule of layer-wise relevance propagation (LRP), and related methods, while DeepLIFT avoided the collapse. If this theory transfers to clinical 3D networks, the non-negative methods should show substantial raw agreement with pathology but near-zero learned agreement. DeepLIFT and perturbation methods should keep a positive learned agreement.

Relation to Prior Work

The closest work is Explainable AI for CNN-based prostate tumor segmentation in multi-parametric MRI correlated to whole mount histopathology. That study trained a 3D U-Net on multi-parametric MRI with radiologist contours and ran a cascaded randomization check on 3D Grad-CAM by visual inspection. It then found that Grad-CAM correlated better with registered whole-mount pathology than the binary segmentation did, on 15 test patients. That study supplies our setting and our main baseline protocol. It differs from ours in three ways. It treated randomization and histological agreement as two separate checks rather than computing their difference. It tested only Grad-CAM. It reported no numerical randomization score. Our proposal turns its qualitative finding, that the heatmap finds the lesion core, into a testable decomposition. That decomposition could show the finding is partly or wholly explained by input structure.

The randomization methodology comes from three papers. Sanity Checks for Saliency Maps introduced the model parameter randomization test (MPRT) and showed that many maps survive weight destruction. A Fresh Look at Sanity Checks for Saliency Maps contributed two refinements. Smooth MPRT averages attributions over noisy inputs so that high-frequency gradient noise no longer fakes sensitivity, and efficient MPRT measures the rise in map entropy instead of pairwise similarity. The same paper argued that bottom-up randomization is the stricter direction. We reuse these tools. Their own open problem, that conclusions shift with the choice of similarity metric (see also Sanity Checks for Saliency Metrics), also motivates us. Pathology offers an external reference against which the internal sanity-check metrics can be compared. We treat it as an anchor rather than an oracle, because pathology does not define faithfulness.

xMIL: Insightful Explanations for Multiple Instance Learning in Histopathology is the main signed-evidence precedent. That work showed that attention weights over slide patches are unfaithful and blind to evidence against a diagnosis. It used a modified LRP that keeps signed relevance and evaluated faithfulness by patch removal. It operates only on slide images and never ran parameter randomization. We do not work on slides. We borrow its argument that negative evidence is clinically meaningful and test whether signed attributions on MRI place negative evidence in histologically benign gland tissue.

The validation side rests on two papers. Bridging the gap between prostate radiology and pathology through machine learning showed that radiologist outlines miss about 25% of cancer volume and that models trained on pixel-level digital pathology labels detect cancer better. Weakly Supervised Registration of Prostate MRI and Histopathology Images quantified how well histology can be aligned to MRI: Dice 0.90 internally and 0.89 externally, with Hausdorff distances of roughly 4.4 to 5.0 mm. These papers constrain the design. The registration error is large relative to small lesions, so any agreement difference we report has to exceed the change that misregistration alone would cause.

Research Program

We will train two 3D U-Nets of the Gunashekar design on T2-weighted, diffusion coefficient, and high b-value inputs. The two models are identical except for their labels. One is trained on radiologist contours and one on pixel-level pathology labels, mirroring the label contrast in the Stanford surgical cohort used by "Bridging the gap." If that cohort cannot be obtained, we fall back to the public TCIA Prostate Fused-MRI-Pathology and Prostate-MRI sets, which contain 16 and 25 histology-paired patients. In that case we create a simulated radiologist label by removing the lowest-conspicuity 25% of each pathology lesion. This fallback is a weaker, controlled stand-in, and we will label it as such.

For each model we compute a scalar lesion score, the summed tumor logit inside the predicted gland. We explain that score with eight methods. These are Grad-CAM, raw gradient, SmoothGrad, Integrated Gradients, DeepLIFT, z+ LRP, Guided Backpropagation, and occlusion-style perturbation in the spirit of Fong et al.. Each method is computed on the trained weights and on every stage of top-down and bottom-up cascaded randomization, with five seeds per stage.

Three analyses follow. The first estimates $\Delta_H$ for every method and model, both over the whole gland and over the "missed region," meaning pathology-positive voxels outside the radiologist contour. The second handles registration uncertainty. We redraw the histology-to-MRI alignment under random smooth deformations, calibrated so that boundary error matches the reported Hausdorff distances, and recompute $\Delta_H$. A difference counts only if it exceeds the 95% envelope that these deformations produce. This directly addresses the registration-confound gap: a drop in agreement caused by warping is separated from a drop caused by the model. The third computes smooth MPRT and efficient MPRT scores for the same methods and tests whether their rankings agree with the ranking by $\Delta_H$. Where they disagree, we report which internal metric tracks the external anchor.

For the signed methods (gradient, Integrated Gradients, DeepLIFT), we additionally measure whether negative attribution concentrates in benign gland tissue that looks dark on diffusion imaging. This is a mimic region where evidence against cancer should be informative.

Baselines

The naive reference consists of maps that use no trained network at all. These are the inverted diffusion-coefficient intensity, the 3D Sobel edge magnitude of the stacked inputs, and each attribution method applied to a fully randomized network. If any of these matches a trained method's raw agreement with pathology, raw agreement cannot be used as validation. They are computed on exactly the same voxels and the same registration draws as every other method.

The strongest existing method is the Gunashekar protocol: 3D Grad-CAM on a radiologist-trained U-Net, evaluated by correlation with registered whole-mount pathology and backed by a visual cascaded-randomization check. The paper reports a tumor Dice of about 0.31 against pathology and 0.32 against radiologist contours, with inter-observer Dice of 0.48–0.52. It gives no numerical randomization score, so we found no reported result on Lacuna for a quantitative parameter-sensitivity score of Grad-CAM on this task. We will reimplement the protocol on our cohort with the same network, data split, and compute. The paper's 0.31 tumor Dice will serve as a check that our model is comparable.

Beating this baseline does not mean higher raw overlap. It means showing that $\Delta_H$ ranks methods differently from raw overlap, outside the registration envelope. In particular, at least one method with high raw agreement must have a learned component indistinguishable from zero, or Grad-CAM's apparent advantage must turn out to be mostly input-driven.

The model's own soft probability map serves as the reference for localization. An ablation that removes our key ingredient, the randomization control, reports raw $\mathrm{AUC}_H$ alone. This shows how many conclusions change when the control is added. A second ablation restricts randomization to the top-down direction only, which tests whether the bottom-up direction recommended by "A Fresh Look" matters in practice.

Why not just…?

Why not just run the modern sanity checks without pathology? Smooth and efficient MPRT tell us whether a map depends on the weights, not whether what it highlights is biologically meaningful. They also disagree with each other: the same paper found Guided Backpropagation looked insensitive under similarity-based checks but sensitive under the entropy-based one. Without an external reference there is no way to settle which verdict to trust on clinical data. Our third analysis tests whether either of them tracks the pathology-anchored quantity.

Why not just measure overlap with pathology, as Gunashekar et al. did? Overlap alone rewards anything that follows tumor appearance. Tumors are dark on diffusion imaging, and rank-1 collapse theory predicts that non-negative methods return input-determined maps. The randomized-network and Sobel baselines test this directly. If they reach raw agreement close to the trained methods, overlap-based validation is shown to be insufficient on this exact task.

Why not just compare the model's soft output to pathology and skip attribution? This is a real threat for localization, since attribution of a lesion score may largely reproduce the output map. We report the soft output as a reference. If every attribution method's learned agreement falls at or below what the output map achieves, and the missed region shows nothing beyond the output, we will conclude that attribution adds nothing for localization in this setting. That would be a useful negative result for a field that routinely presents heatmaps as validation. The question attribution can still answer is whether the radiologist-trained model shows evidence on missed cancer that its own thresholded output discards, which Gunashekar's lesion-core observation suggests.

Decisive Checks

The proposal succeeds if three things hold. First, the learned histological agreement separates the method families predicted by rank-1 theory, outside the registration envelope. Second, the method ranking under $\Delta_H$ differs from the ranking under raw overlap. Third, in the missed region the pathology-trained model shows higher learned agreement than the radiologist-trained model, while the input-only baselines show no such gap.

The idea is falsified in two ways. If all methods, including Guided Backpropagation and z+ LRP, keep large learned agreement in clinical 3D networks, rank-1 collapse does not transfer to this setting. If registration perturbations swamp every difference, cohorts of this size cannot support pathology-anchored attribution claims, and the field needs to know that too. Statistics will use patient-level bootstrap, since the voxels within a patient are not independent. With 15 to 40 test patients, we will pre-register the primary comparison, DeepLIFT versus Guided Backpropagation $\Delta_H$ on the pathology-trained model, to avoid multiple-comparison inflation.