Medical image classifiers are often explained with heatmaps. For tiny findings such as a microcalcification of a few pixels or a small lung nodule, a heatmap can look precise without meaning that the model actually resolved the finding. It may have inferred "cancer" from the tissue around it, at coarse scale, and the map simply looks sharp. This proposal separates two things that current work treats as one. Pixel resolution is how fine the map looks. Information resolution is how fine the image detail was that the decision depended on. We want to measure the second directly.
The approach uses high-resolution backbones in the style of HRNet. These are networks that keep a full-resolution feature stream alive through their whole depth, running in parallel with half-, quarter- and eighth-resolution streams that repeatedly exchange information. In such a network the evidence for a decision travels along identifiable scale pathways. We place a learned information bottleneck on each stream at every exchange point. A bottleneck here is a per-location mask that replaces features with noise wherever they are not needed for the prediction. This yields one attribution map per scale, and with it a causal statement of the form: "at this location the prediction needed information that only the full-resolution stream carried." The clinical payoff is an audit check that distinguishes "the model resolved the microcalcification" from "the model guessed from context." That distinction matters for trust in exactly the tiny-lesion cases where current maps are least reliable.
Core Claim
An explanation cannot be finer than the pathway its evidence travelled through. Today's high-resolution explanations do not check this. Multi-layer Information Bottleneck Attribution (Explaining COVID-19 and Thoracic Pathology Model Predictions by Identifying Informative Input Features) fuses bottleneck masks from shallow and deep layers of an ordinary downsampling CNN, which doubles box overlap over Grad-CAM. But the shallow layers in that network feed every later layer, and their sharp masks say nothing about whether fine detail survived the later pooling. That paper motivates this work by showing resolution matters for attribution. It also leaves the question open, because the backbone itself still discards detail.
A naive fix would be to run Grad-CAM on HRNet's final full-resolution output (Deep High-Resolution Representation Learning for Human Pose Estimation). This has the same flaw in a different form. HRNet fills its high-resolution stream partly by nearest-neighbour upsampling of the coarse streams, so a full-resolution map can contain coarse information disguised as fine. Our claim is that per-stream bottlenecks expose this. The fraction of decision information routed through the full-resolution stream inside a lesion should predict whether the decision really depends on the lesion's fine morphology. We test this against an intervention that does not rely on any attribution method.
Research Program
We train HRNet-W32 classifiers with an image-level head on chest radiographs and mammograms. At each of HRNet's cross-scale exchange units we attach a mask $\lambda_s(x)\in[0,1]$ for every stream $s$ and location $x$. The masked feature is $z_s = \lambda_s h_s + (1-\lambda_s)\epsilon_s$, where $h_s$ is the original feature and $\epsilon_s$ is Gaussian noise matched to that feature's statistics. This mixing scheme follows the bottleneck idea of the IBA work above. For each image we fit the masks to
$$\min_{\lambda}\; -\log p(y \mid z_\lambda) \;+\; \beta \sum_{s} \sum_{x} \mathrm{KL}\big(p(z_s(x)\mid h_s(x)) \,\|\, q(z_s(x))\big),$$
where $y$ is the predicted label, $q$ is the noise prior, and $\beta$ trades prediction retention against information passed. We also fit a deletion form, in the spirit of the paper's Inverse IBA, that recovers all informative regions rather than a sufficient subset. Missing a second lesion is the clinically costly error.
The output has two parts. The first is a stack of per-scale maps. The second is a scale profile for any annotated region: the share of retained information that passed through each stream. A lesion whose profile is dominated by the full-resolution stream is one the model "saw." A lesion whose profile sits in the 1/8 stream is one the model inferred from its surroundings.
The interpretive bet needs ground truth that does not come from attribution. We supply it with a low-pass swap. Inside each annotated lesion footprint we replace the image with a Gaussian-blurred copy. This removes fine morphology while preserving local mean intensity and surrounding context, and blurring cannot add new structure, which is the scale-space causality property used by Attribution in Scale and Space. If the prediction changes, the decision depended on fine detail at that lesion. If it does not, it did not. The swap gives a per-lesion binary label of "fine-detail dependent." An honest scale profile should predict that label.
A second ground truth comes from planted shortcuts, extending the shortcut audits of Improving Explainability of Disentangled Representations using Multipath-Attribution Mappings. That work found trimcut borders and lesion-size shortcuts with a disentangled autoencoder whose global encoder cannot see tiny cues. We train controlled variants in which the label is perfectly predicted by either a 4–8 pixel marker or a large low-frequency intensity gradient. The correct scale and location of the evidence are then known by construction.
Data and Setting
NIH Chest X-ray8 is the main attribution benchmark because it has radiologist boxes and a reported baseline. INBreast is the tiny-target regime, with microcalcifications of at most 14 pixels and often fewer than 5, as characterised in Cascaded Generative and Discriminative Learning for Microcalcification Detection in Breast Mammograms. Its pixel annotations make the low-pass swap and pointing tests possible. INBreast is small, so we initialise from the chest-trained network and report variance across folds.
A 3D extension on LUNA16 nodules is a stretch goal. Memory limits make it uncertain: Whole brain segmentation with full volume neural network already trains a 3D HRNet at batch size 1, and adding per-stream masks raises memory further. We would attempt it only if the 2D results hold.
Baselines
The do-nothing reference is Grad-CAM on a ResNet-50 classifier trained on the same data. On Chest X-ray8 its reported mean IoU against expert boxes is 0.077 (Explaining COVID-19 and Thoracic Pathology Model Predictions by Identifying Informative Input Features). This is the map clinicians are most often shown.
The strongest reported method is Multi-layer IBA on a ResNet-50, at 0.189 mean IoU on Chest X-ray8 in the same paper. We rerun it on our own trained ResNet so that classifier quality is matched. We give it the same number of per-image optimisation steps as our method and tune its $\beta$ on the same validation split. Beating it requires two things. On box IoU we must be at least comparable, since box IoU rewards coarse overlap and is not our target. On the low-pass swap test we must be clearly better: the AUROC with which the attribution, restricted to a lesion, predicts whether the swap flips the decision.
The strongest simple competitor is Grad-CAM and plain single-mask IBA computed on HRNet's fused full-resolution output. It shares our backbone, compute and pixel resolution. It is the direct test of whether pixel resolution alone suffices.
We also run BlurIG (Attribution in Scale and Space) on both backbones, because it is the established scale-aware attribution. Where engineering allows, we run a GLAM-style global-then-local patch pipeline (Weakly-supervised High-resolution Segmentation of Mammography Images for Breast Cancer Diagnosis) on INBreast. It reports relative Dice gains of 39.6% for malignant lesions over CAM-based methods on the NYU screening data, but we found no reported result on Lacuna for it on INBreast or Chest X-ray8.
The key ablation keeps everything about our method fixed but uses a single mask on the fused full-resolution representation instead of per-stream masks. This isolates whether the decomposition by stream, rather than the backbone, carries the gain. All classifiers must reach image-level AUC within one point of each other, so that attribution differences are not differences in model quality.
Why not just…?
Why not just use Multi-layer IBA on an ordinary CNN, since it already produces crisp maps? Its crispness comes from early-layer masks in a network that later pools that detail away. It can therefore highlight a microcalcification that the classifier never resolved. The low-pass swap is designed to catch exactly this. If Multi-layer IBA predicts swap outcomes as well as our per-stream profiles do, we will report that and conclude that the backbone does not matter for faithfulness.
Why not just run Grad-CAM on HRNet's full-resolution output? This is the most serious threat. It is cheap and already high-resolution. Our argument is that upsampled coarse features fill that grid, so its maps should look sharp yet fail to tell "saw it" from "inferred it." The single-mask ablation and the planted 4–8 pixel shortcut settle this. If Grad-CAM on HRNet localises the tiny marker and predicts swap flips as well as per-stream masks do, the decomposition is unnecessary. The contribution then shrinks to the evaluation protocol, which we would still release.
Why not just use BlurIG, which is already scale-aware? BlurIG varies scale at the input and reports which input frequencies mattered. It does not report whether the network's internal path preserved them. It is a natural partner: agreement between BlurIG's dominant scale and our stream profile is itself a consistency check, and disagreement on lesions that pass the swap test is diagnostic.
Why not just crop high-resolution patches, as GLAM does? Patch pipelines produce fine maps of whatever the coarse first stage selects. GLAM's authors note that lesions missed at the coarse stage never reach the local model. Such pipelines also cannot say whether the global decision used the patch detail. We include the GLAM-style pipeline on INBreast to measure how often swap-dependent lesions go unselected.
Decisive Checks
The central experiment is the low-pass swap AUROC on INBreast microcalcifications and Chest X-ray8 nodule and mass boxes, computed for every baseline. The expected result is that all HRNet-based maps match on box IoU. If the core claim is right, only the per-stream profile separates swap-dependent lesions from swap-independent ones. A clear failure of that prediction refutes the core claim.
The planted-shortcut models give an unambiguous second test. The method must attribute the tiny-marker model to the full-resolution stream at the marker. It must attribute the gradient model to the coarse stream across the image. Both maps must change when the network's weights are randomised; a map that survives randomisation is not explaining the model.
Finally, the placebo finding of Challenging common interpretability assumptions in feature attribution explanations matters here: users chose the better model 60.2% of the time with random-noise explanations, versus 61.6% with SHAP. In light of that, we run a small reader study with clinicians. They judge whether a model truly detected a lesion, shown either the scale profile, a single HRNet heatmap, or a random map of matched sharpness. The question is whether the profile helps beyond visual plausibility. A null result here would be reported as a limit on the clinical value of the method, even if the automated tests succeed.
References
- Explaining COVID-19 and Thoracic Pathology Model Predictions by Identifying Informative Input Features
- Deep High-Resolution Representation Learning for Human Pose Estimation
- Attribution in Scale and Space
- Improving Explainability of Disentangled Representations using Multipath-Attribution Mappings
- Cascaded Generative and Discriminative Learning for Microcalcification Detection in Breast Mammograms
- Whole Brain Segmentation with Full Volume Neural Network
- Weakly-supervised High-resolution Segmentation of Mammography Images for Breast Cancer Diagnosis
- Challenging common interpretability assumptions in feature attribution explanations