When a medical segmentation model trained on one imaging modality, such as cardiac MRI, is moved to another, such as CT, the usual repair is self-training. The model labels the unannotated target scans itself, the labels it is most confident about are kept as "pseudo-labels," and it is retrained on them. The weak point is the filter. Confidence measures how sure the model is, not why it is sure, and under a modality shift a model can be confidently right or confidently wrong for reasons unrelated to anatomy. Examples include a new contrast profile, a scanner artifact, or a neighbouring bright structure. We propose to filter pseudo-labels by asking a different question: does the model support this target-modality prediction with the same kind of spatial evidence it used when it was correct on the source modality?
The evidence comes from Grad-CAM (gradient-weighted class activation mapping). Grad-CAM is a post-hoc heatmap method. It weights a convolutional layer's feature maps by how much each one raises a chosen output score, which highlights the image regions the prediction depends on. We express each heatmap relative to the structure the model predicted, fit a one-class "normal evidence" model on source-domain predictions that were verified correct, and reject target pseudo-labels whose evidence looks anomalous. Evidence in this note means the evidence pattern, the predicted-structure-relative Grad-CAM map. The filter needs one backward pass per candidate and no target annotations. It plugs into existing cross-modality pipelines on MM-WHS cardiac MRI↔CT and on retinal fundus-to-OCT-A vessel segmentation.
Core Claim
Our claim is that, under a modality shift, how a prediction is supported carries information about pseudo-label correctness that confidence and augmentation stability do not. Using that information to filter self-training data should improve unsupervised cross-modality adaptation. We expect the largest gains in the hard direction, CT→MRI, where existing methods trail the reverse direction by 11 Dice points.
This differs from the closest work in three ways. Semi-supervised Image Classification with Grad-CAM Consistency adds an L2 penalty between Grad-CAM maps of an image and its geometric augmentation on CIFAR-10. That method needs a known pixel correspondence between the two views, which unpaired MRI and CT scans never have. Expert-Guided Explainable Few-Shot Learning aligns Grad-CAM to radiologist regions of interest and selects active-learning samples by "explanation misalignment." Both steps need expert masks, and its "cross-modality" result on breast ultrasound comes from retraining on that modality rather than transferring to it. In OCT retinopathy sub-domain adaptation, Grad-CAM appears only as a retrospective picture after pseudo-label adaptation, and the shift studied is within one modality. The method we borrow most directly is from automated processing of XAI outputs for infrastructure fault diagnosis. That work fits Deep SAD, a semi-supervised one-class anomaly detector, on Grad-CAM heatmaps to flag predictions that lean on shortcuts such as component ID tags. We move this idea from a single domain into the cross-modality gap and use it as a training-data filter rather than a review aid.
Research Program
Computing the evidence map. For a segmentation network and structure class $c$, we take the target score $y^c$ to be the sum of class-$c$ logits over the predicted mask $\hat{m}_c$. We compute 3D Grad-CAM at the bottleneck and at one higher-resolution decoder layer. The 3D version averages gradients over depth, height and width, following AID-Net. The second layer is there because stride-32 maps are too coarse for thin myocardium or microvessels.
Making the map comparable across modalities. Each map is resampled into a normalized frame centred on the predicted structure's bounding box and scaled to it, with a fixed margin of surrounding context. A heatmap from CT and one from MRI therefore become comparable descriptions of where, relative to the predicted ventricle, the model looked.
Fitting the evidence model. On source-domain validation scans, the predictions with Dice above a threshold count as "normal evidence." Low-Dice predictions are used as labelled anomalies, which is what makes Deep SAD semi-supervised. A small Deep SAD encoder is fit per structure on these maps.
Scoring and filtering. For a target candidate $x$ with class $c$, the evidence score is
$$s_c(x) = \lVert \phi_c(\tilde{L}_c(x)) - z_c \rVert^2,$$
where $\tilde{L}_c(x)$ is the mask-normalized Grad-CAM map, $\phi_c$ is the Deep SAD encoder, and $z_c$ is its learned centre. The confidence filter is left in place, and a candidate becomes a pseudo-label only if it passes that filter and its $s_c$ falls below a quantile threshold. The threshold is chosen on source data only.
Analysis program. In parallel we test the asymmetry hypothesis left unverified by Structure-Driven UDA. The hypothesis is that CT→MRI is harder because, on MRI, the model's evidence spreads into surrounding soft tissue. If it holds, the evidence-score distribution on the target should shift more in CT→MRI than in MRI→CT, and the shift should track per-structure Dice loss.
Baselines
Naive references. The floor is the source model applied without adaptation. Structure-Driven UDA reports 64.9% mean Dice for MRI→CT and 38.1% for CT→MRI on MM-WHS without adaptation. The second reference is plain self-training with a confidence threshold, run on the same adapted starting model as every other variant. This is the direct competitor the filter must beat.
Strongest existing method. The strongest baseline on MM-WHS is Structure-Driven UDA, which combines self-supervised 3D anatomical landmarks with adversarially aligned Canny edges. It reports 85.8% mean Dice for MRI→CT and 74.8% for CT→MRI. We use it as the starting model and add our filtered self-training on top, so the question is whether vetting adds Dice beyond a strong aligner rather than whether it replaces one. Because the file gives no code resource, we will reimplement it and report our reproduction next to the published numbers. Gains are measured against our reproduction. As a weaker aligner, we also run the adversarial domain-adaptation-module approach of Unsupervised Cross-Modality Domain Adaptation of ConvNets, which reports 57.2% average Dice for MRI→CT against a 14.3% unadapted baseline.
For retina, the strongest baseline is AdaptDiff, which reports 81.94% Dice on OCTA-500(6M) against 73.16% without adaptation and an 84.78% supervised oracle. AdaptDiff trains its conditional diffusion generator on noisy target pseudo-labels, so our filter applies naturally to its pseudo-label stage and leaves the rest of the pipeline unchanged.
Fairness and what counts as beating them. All self-training variants share one starting model, the same unlabeled target pool, the same number of rounds, and the same retained fraction of candidates. Fixing the retained fraction rules out gains that come only from keeping fewer labels. Every threshold is tuned on source data with an equal budget. Beating a baseline means a gain in mean target Dice over three seeds that exceeds the seed spread, with the CT→MRI gain at least as large as the MRI→CT gain.
Ablations isolating the key ingredient. The first ablation replaces Grad-CAM with plain activation magnitude, which keeps the evidence model but drops the gradient weighting. The second removes mask-relative normalization and uses raw image coordinates. The third swaps Deep SAD for a fixed rule, the fraction of attribution mass inside the dilated predicted mask. Together these show whether the cross-modality evidence comparison matters, or whether any heatmap statistic would do.
Why not just…?
Why not just tighten the softmax confidence threshold? Confidence measures certainty, not its cause. The infrastructure study here found confidently correct predictions resting on ID tags. EGxFSL reports that sample selection by explanation misalignment reached 76.85% accuracy on BraTS with 680 labels, versus 58.01% for random sampling. Neither result is cross-modality or segmentation, so this remains an assumption for our setting. We settle it directly by computing, on held-out target labels used for analysis only, the AUROC of each score for detecting bad pseudo-labels. Confidence thresholds are swept at matched retention.
Why not just keep pseudo-labels that are stable under augmentation or dropout? This is the real threat. Stability is cheap, and it may already capture whatever Grad-CAM detects. The CIFAR-10 evidence that attribution adds to output consistency is small: a 0.31 ± 0.59 point average gain in Grad-CAM consistency. We therefore make a decisive test. Combining the evidence score with stability must improve bad-label AUROC over stability alone, and the partial correlation of $s_c$ with pseudo-label error must stay clearly non-zero after controlling for stability and confidence. If it does not, we report the method as redundant.
Why not just use better alignment, such as Canny-edge or landmark structure guidance? These approaches act on representations, while ours acts on which target data are trusted. The two are complementary, which is why Structure-Driven UDA is our starting model rather than a rival. Raw edges are not a substitute. The structure-driven paper had to pair Canny edges with learned landmarks because edges fire on ribs and artifacts without regard to class.
Why not just translate target images into the source style? AdaptDiff shows that CycleGAN-style translation erases microvessels, sometimes dropping below the unadapted baseline. Its diffusion alternative still depends on pseudo-labels, which is exactly the stage our filter targets.
Proposed Experiments
The main experiment runs MM-WHS in both directions over four structures: ascending aorta, left atrium, left-ventricle blood pool and myocardium. A second experiment uses the retinal suite with fundus photography as source and OCTA-500 as target, through AdaptDiff's pseudo-label stage. Metrics are per-structure and mean Dice, bad-label AUROC for each filter, and wall-clock overhead per scored candidate.
Three checks would decide the question. First, the evidence score must beat confidence and augmentation stability at detecting wrong pseudo-labels, and must add to them in combination. Second, filtered self-training must improve on our reproduction of Structure-Driven UDA in the CT→MRI direction at matched retention, and must improve AdaptDiff's OCTA-500 Dice toward the 84.78% oracle. Third, the asymmetry analysis must show a larger shift in evidence scores for CT→MRI that correlates with per-structure Dice loss.
If the first check fails, the idea is wrong, and the other results would not rescue it. A clean negative result would still be useful: it would show that Grad-CAM evidence under modality shift carries no information beyond what the model's outputs already reveal.
References
- Semi-supervised Image Classification with Grad-CAM Consistency
- Expert-Guided Explainable Few-Shot Learning with Active Sample Selection for Medical Image Analysis
- OCT Retinopathy Classification via a Semi-Supervised Pseudo-Label Sub-Domain Adaptation and Fine-Tuning Method
- Automated processing of eXplainable Artificial Intelligence outputs in deep learning models for fault diagnostics of large infrastructures
- Coronary calcium detection using 3D attention identical dual deep network based on weakly supervised learning
- Structure-Driven Unsupervised Domain Adaptation for Cross-Modality Cardiac Segmentation
- Unsupervised Cross-Modality Domain Adaptation of ConvNets for Biomedical Image Segmentations with Adversarial Loss
- AdaptDiff: Cross-Modality Domain Adaptation via Weak Conditional Semantic Diffusion for Retinal Vessel Segmentation