Which Hot Spot Is Cancer? Cross-Modal Attribution for Weakly Supervised PET/CT Lesion Masks

Whole-body FDG-PET/CT scans pair a PET image, which shows where a radioactive sugar tracer accumulates (metabolically active tissue), with a CT image of the anatomy. Tumors light up on PET. So do the brain, heart, kidneys and bladder, which burn or excrete sugar normally. Segmentation models that outline tumors in these scans currently need radiologists to trace every lesion voxel by voxel. The autoPET challenge shows that such models reach about 0.8 Dice, and also shows how expensive that tracing is. Hospital archives, by contrast, hold large numbers of scans whose only label is whether the patient has cancer. This proposal asks whether that one bit per scan is enough to build training masks.

The idea separates two jobs. The first is finding the shape of each candidate lesion. PET physics already does this well: regions of elevated uptake have sharp, well-defined boundaries. The second job is deciding which hot regions are disease and which are normal physiology. That decision depends on both images at once: high uptake in a place where the CT anatomy says uptake should not occur. We therefore train an early-fusion classifier, meaning a network that receives PET and CT stacked as input channels from the first layer, using only scan-level cancer/healthy labels. We then score each PET-defined candidate region by how much the classifier's cancer prediction depends on the joint PET-and-CT content of that region. That joint dependence is the cross-modal interaction. Candidates with high interaction scores become pseudo-masks, and a standard segmentation network is trained on them.

Core Claim

In FDG-PET/CT, the attribution signal that separates lesions from physiological uptake is the cross-modal interaction term, not the total saliency. Total saliency is dominated by the PET main effect, which is "this is hot" and fires on the bladder as readily as on a metastasis. The claim makes three testable predictions. First, ranking candidates by region-level interaction will select lesions more accurately than ranking them by standard Grad-CAM overlap, by PET-only attribution, or by the instance scores of a multiple-instance classifier. Second, the advantage will be concentrated in organs with high physiological uptake. Third, the advantage will mostly vanish on brain MRI, where one sequence carries most of the signal.

This design avoids the two failure modes the saliency literature keeps reporting for weakly supervised masks. Grad-CAM-style maps are coarse and "peaky": they cover only the most discriminative core of an object. AME-CAM attacks this with multi-exit maps, and the histopathology work using vision-language fusion names peakiness as its main failure. Information-bottleneck attribution has the opposite problem: IBA explicitly reports underestimating lesion extent. In our pipeline, attribution never draws a boundary. It only casts a vote per candidate region. The boundary comes from the uptake itself.

Relation to Prior Work

Two weakly supervised CAM methods motivate the approach and define its closest neighbors. AME-CAM stacks four MRI sequences and aggregates multi-exit class activation maps on 2D BraTS slices. It never asks which sequence drove an activation. C-CAM removes co-occurring-organ confounders using a fixed anatomical layout prior. That prior constrains our design: metastases appear anywhere, so a layout prior is not available, and our "anatomy context" must instead be learned by the classifier through the CT channel.

InterSHAP supplies the method. It decomposes model behavior into per-modality main effects and a Shapley interaction term, and it shows early fusion produces far more interaction than late fusion. It does this only at the whole-input level, on tabular and EHR data, and warns of exponential cost in the number of modalities. With two modalities the cost is four forward passes. Making the decomposition region-local is therefore cheap, and that region-local version is our addition.

M2IB also uses one modality to explain another. It does so for image–text pairs, as a post-hoc explanation tool costing about 100 optimization steps per sample, and it does not produce training masks.

Reports that gradients in multimodal models are dominated by one modality (When One Modality Rules Them All; Can We Talk Models Into Seeing the World Differently?) motivate our prediction that total saliency will track PET alone. UniMiSS shows that 3D self-supervised pretraining reduces dense-label needs. It is evaluated only with dense supervision, so we treat it as an optional backbone ablation rather than as a competitor.

Research Program

Candidate generation. Each training scan is converted into PET-defined candidate regions. We take connected components above an uptake threshold (standardized uptake value, the usual normalized PET intensity), using a few thresholds so that small and faint lesions are not lost. Components inside the brain and bladder are kept rather than discarded, because the experiment needs the classifier to learn to reject them.

Weak classifier. A 3D early-fusion classifier is trained on all autoPET training scans with only the scan-level label. Healthy controls are negatives; lymphoma, lung cancer and melanoma studies are positives. Ground-truth masks are hidden during this stage.

Region interaction score. For each candidate region $r$, let $f(x)$ be the classifier's cancer logit on scan $x$. Let $x^{\text{P}}_r$ be the scan with PET inside $r$ replaced by a local background uptake level, $x^{\text{C}}_r$ the scan with CT inside $r$ replaced by an inpainted version of the surrounding tissue, and $x^{\text{PC}}_r$ the scan with both replacements. The region-level interaction is

$$I(r) = f(x) - f(x^{\text{P}}_r) - f(x^{\text{C}}_r) + f(x^{\text{PC}}_r),$$

which is the two-player Shapley interaction restricted to region $r$. The corresponding PET main effect, $f(x) - f(x^{\text{P}}_r)$ averaged over the CT state, is computed from the same four passes.

Gradient variant. Because the replacement baselines are a real modeling choice and can create out-of-distribution inputs, we also compute a gradient-only version. It is per-channel integrated gradients summed over $r$, with a cross-channel second-order term estimated along the integration path. Integrated gradients is chosen over vanilla gradients specifically to address the saturation problem. We report whether the cheap gradient version tracks the occlusion version.

From scores to masks. Candidates are thresholded on $I(r)$, with the threshold calibrated so that healthy-control scans yield near-zero selected volume. Selected candidates form the pseudo-masks. An nnU-Net, the self-configuring 3D U-Net used by top autoPET teams, is then trained on these pseudo-masks over the full cohort.

Ablation. A late-fusion version replaces the early-fusion classifier with separate PET and CT classifiers whose outputs are averaged. This tests whether the early-fusion ingredient matters.

Baselines

The naive reference point is the candidate generator alone. Every uptake component above threshold is labeled lesion, optionally after removing a fixed list of high-uptake organs found by an off-the-shelf CT organ segmenter. It requires no learning and no labels. It is the baseline a nuclear-medicine reader would propose first. If our method does not clearly beat it on Dice and false-positive volume, the classifier has contributed nothing. No reported result was found on Lacuna for uptake thresholding on autoPET, so we will run it ourselves on identical folds.

The strongest weakly supervised competitor is multiple-instance learning over the same candidates. A network scores each candidate crop (PET+CT), pools the scores to a scan prediction, and is trained on the same scan labels. Its instance scores are a direct alternative selection signal that needs no saliency at all, which makes it the most dangerous competitor. It gets the same candidates, backbone family, training data, and hyperparameter search budget as our classifier. We also run AME-CAM-style multi-exit CAM extended to 3D, and plain Grad-CAM thresholding, both trained on the identical early-fused input. These show what direct saliency-to-mask pipelines achieve when attribution draws the boundary. No reported result was found on Lacuna for any weakly supervised method on autoPET.

The upper reference is a fully supervised nnU-Net. The autoPET challenge reports mean Dice 0.79 for the top single team and 0.81 for the ensemble, false-negative volume between 0.5 and 1.5 ml, and 0.88 internal versus 0.6–0.7 external Dice. We retrain it on our cross-validation folds so that numbers are comparable, rather than quoting the challenge test-set figures as if they were on our split.

Isolating the key ingredient. The central ablation scores the same candidates by total attribution and by PET main effect instead of interaction. The claim is falsified if interaction does not beat both. Beating MIL by a margin larger than fold-to-fold variation, at matched healthy-scan false-positive volume, counts as a win. Matching MIL counts as a negative result for saliency.

Why not just…?

Why not just threshold the PET uptake? Thresholding gets boundaries roughly right, which is why we reuse it for shape. It cannot tell a metastasis from myocardium or a full bladder. The autoPET report itself documents physiological uptake in brain, heart, kidneys and bladder. The naive baseline above measures exactly how much false-positive volume thresholding leaves. If organ masking alone closes the gap, the classifier is unnecessary and we will say so.

Why not just use multiple-instance learning on the candidates? This is the real threat. MIL instance scores may already encode "hot but anatomically wrong" without any attribution step. Our argument is that a crop-level instance classifier sees limited anatomical context, while a whole-scan early-fusion classifier can use global anatomy, such as where the bladder is in this patient. That argument may be wrong. The head-to-head comparison at matched budgets settles it. If MIL ties, the contribution shrinks to an analysis of why interaction and MIL scores agree.

Why not just annotate 10% of scans densely? With strong pretraining this is a sound approach, and we treat it as an empirical question. We train nnU-Net on $k$ densely labeled scans for increasing $k$ and report the $k$ at which it matches our weakly trained model. The weak pipeline can use every scan in a multi-center archive, but whether that breadth helps under domain shift has to be measured, not assumed. We probe it by holding out cancer subtypes. The challenge's external drop to 0.6–0.7 Dice suggests that breadth matters.

Why not just use RECIST diameters or a lesion-healing generative model? RECIST lines (lesion co-segmentation) still require a radiologist's measurement per lesion. Archival scans with only a diagnosis do not have them, and they are defined on CT, not PET. PathoGAN-style subtraction would need to synthesize a healthy whole-body PET/CT despite large normal variation in uptake between patients. We include RECIST only as a cost comparison, not a baseline, because autoPET does not provide the lines.

Proposed Experiments

The primary benchmark is autoPET. We use five-fold patient-level cross-validation on the 1,014 training studies, with dense masks used only for scoring. Metrics are Dice, false-negative volume and false-positive volume, which follow the challenge, plus per-candidate selection precision and recall broken down by organ.

The first decisive check is the organ breakdown. If the interaction advantage over PET main effect appears in kidney, bladder and heart regions and is near zero in lung and lymph-node regions, the mechanism is as claimed. A uniform advantage would suggest a generic regularization effect.

The second check is BraTS 2021 as a control domain. There, AME-CAM reports Dice 0.862 on T2-FLAIR with image-level labels, against about 0.91 for full supervision. Brain tumors lack a physiological look-alike of comparable salience, so we predict interaction scoring gives little gain over total attribution. A large gain there would mean we have misidentified the reason the method works.

The third check is cost and faithfulness. We measure agreement between the four-pass occlusion interaction and the integrated-gradient approximation, and how robust selection is to the choice of replacement baseline. A method whose rankings flip when the CT inpainting changes is not usable.

Secondary analyses are the late-fusion ablation and swapping in a self-supervised 3D backbone in the spirit of UniMiSS. Both ask whether the classifier's interaction signal, not the candidate generator, is what drives the final segmentation quality.