A chest CT or knee MRI usually shows several findings at once, and many of them travel together. A classifier trained to report all of them learns to exploit that. It may call "abnormal" because it saw a torn ligament, or call effusion because it saw an enlarged heart. That is often statistically sensible. It becomes dangerous when a heatmap presents the inference as if the model had seen the second finding directly. Post-hoc heatmaps, meaning saliency maps computed after training by tracing gradients back from an output, cannot tell the two cases apart. They are also known to be unfaithful even in single-label settings. We propose a 3D multi-label classifier whose output for each finding is, by construction, a sum of two parts. The first is evidence the model located in the volume for that finding itself. The second is evidence it borrowed from other findings it located, weighted by a label correlation that depends on the specific scan and on the location in it. The explanation is then an exact readout of the classification head rather than a reconstruction. It says both where the model looked and whether it was seeing or inferring.
The contribution sits where three problems meet. Multi-label learning has shown that label dependencies differ from one instance to the next, and that a single dataset-wide correlation matrix injects noise (Huang et al.). 3D medical models lose spatial context when volumes are handled as disconnected slices. Post-hoc explainers produce plausible but wrong rationales. Our claim is specific. Making label correlation an explicit, spatially localized, per-scan transfer term inside a self-explaining 3D head does three things. It keeps accuracy at the level of an unexplained model. It produces evidence maps that pass deletion and counterfactual tests where post-hoc maps fail. It yields a per-finding "borrowed fraction" that predicts which predictions will break when co-occurrence patterns shift.
Core Claim and Mechanism
The backbone keeps volumetric context in the style of the Medical Slice Transformer (MST). MST encodes each slice with a fine-tuned DINOv2, a 2D self-supervised vision encoder, and then mixes slice embeddings with a transformer that runs along the depth (z) axis. MST reported that its attention localized the lesion core in 57 of 150 blinded cases, against 0 of 150 for Grad-CAM on a 3D ResNet-50. We keep the per-slice patch tokens after inter-slice mixing, so each token $v$ carries out-of-plane context.
On those tokens we place a class-evidence layer following Soft-CAM. Soft-CAM removes global average pooling and instead predicts from spatially pooled class-evidence maps produced by a 1×1 convolution, with sparsity and density penalties. It was shown only in 2D and only with independent class channels. For label $k$ we produce a direct evidence map $E_k(v)$. The label score is
$$ s_k(x) \;=\; \underbrace{\frac{1}{|V|}\sum_{v} E_k(v)}_{d_k:\ \text{direct}} \;+\; \underbrace{\sum_{j\neq k}\frac{1}{|V|}\sum_{v} A_{kj}(x,v)\,\mathrm{relu}\big(E_j(v)\big)}_{b_k:\ \text{borrowed}} $$
The symbols are defined as follows. $x$ is the input volume and $V$ is its set of tokens. $A_{kj}(x,v)\in[0,1]$ is a gate that says how much positive evidence for label $j$ at location $v$ should count toward label $k$ in this scan. We compute it as $A_{kj}(x,v)=\sigma\big(g(h_v, e_k, e_j)\big)$, where $h_v$ is the token feature, $e_k$ and $e_j$ are learned label embeddings, $g$ is a small bilinear network, and $\sigma$ is the logistic function.
This is local label correlation in the sense of feature-conditioned correlation graphs such as Conditional Graphical Lasso and the attention-based label decoder of HOT-VAE. Both condition label interactions on the input. Neither ties the interaction to a spatial location or makes it part of an explanation. The per-finding borrowed fraction $\rho_k = b_k/(|d_k|+b_k)$ is reported alongside every prediction.
The decomposition alone does not stop the backbone from quietly encoding finding $j$'s appearance inside $E_k$. If that happens, correlation leaks into the "direct" channel and the separation becomes cosmetic. We address this with one extra term. The direct score $d_k$ is also trained with binary cross-entropy on a reweighted batch distribution in which each strongly correlated partner $j$ of label $k$ is balanced, so it appears equally often with and without $k$. Under that weighting, partner evidence carries no information about $k$, so $d_k$ is pushed to depend on evidence specific to $k$. The full score $s_k$ is trained on the natural distribution. Whether this term actually removes leakage is an empirical question that the synthetic benchmark below answers directly.
The closest existing work is SEMGNN. It trains a graph network with a label-correlation branch and an edge-mask generator that lets correlated labels share attribution. It reports ground-truth explanation F1 of 0.463 against 0.311 for the GSAT explainer on SynAnchor. We differ in three ways. SEMGNN works on discrete graph edges, while we work on volumetric token grids. SEMGNN encourages correlated labels to share evidence, while we explicitly separate a label's own evidence from evidence it borrows. SEMGNN's masks are an auxiliary output, while our maps are the arithmetic of the prediction head. CT-CLIP handles many co-occurring abnormalities in native 3D, but only through implicit contrastive alignment with text, with no localized evidence. MST supplies the volumetric backbone but addresses single-label tasks.
One constraint shapes the claims. Soft-CAM reports that its most faithful maps did not always match clinician contours. We therefore do not claim our maps will match radiologist annotations better. We claim they are faithful to the head and that they separate the two sources of evidence. Human alignment is measured, not promised.
Research Program
The work has three parts. The first is the model: an MST-style backbone, the 3D direct/borrowed evidence head, and the balanced-partner loss. The model is trained on MRNet and CT-RATE at the depth and in-plane resolution MST used, so the memory budget stays comparable.
The second is a controlled volumetric benchmark with known ground truth, which we call SynVol. It follows the logic of SEMGNN's SynAnchor, but uses real anatomy. We plant synthetic lesion-like textures into LIDC-IDRI or CT-RATE volumes at known locations and generate labels with prescribed co-occurrence. Three kinds of label appear. Some are directly visible. One is defined as "visible finding A is present" and has no visual signature of its own, so it can only be inferred. The last kind is directly visible but correlates with A at 90% in training. Because every rationale is known, we can score whether direct maps land on each label's own lesion and whether the inferred label's evidence is correctly routed through the borrowed channel.
The third is an analysis of whether $\rho_k$ predicts failure under shift. The test sets are subsets in which a finding's usual partner is absent. In CT-RATE these subsets are mined from report labels. In MRNet they are cases with abnormalities but no ACL tear. On SynVol the correlation is simply inverted.
Baselines
The naive reference point is a fine-tuned MST backbone with independent sigmoid heads over pooled features, explained by Grad-CAM and Integrated Gradients. It shows what a practitioner gets today. It is trained on the same splits, with the same slice resolution and the same number of epochs. We expect similar accuracy and worse explanations. Its MRNet reference number is MST's reported AUC of 0.85 against 0.69 for a 3D ResNet-50 MST.
The strongest predictive competitor on CT-RATE is CT-CLIP. It is a native 3D vision transformer pretrained on CT-RATE and reports a zero-shot mean AUROC of 0.731, against 0.629 for the supervised CT-Net CT-CLIP. We found no reported result on Lacuna for its supervised VocabFine fine-tuning. So we fine-tune CT-CLIP ourselves under the same label set, epoch budget, and hyperparameter search size as our model, and explain it with Integrated Gradients. On MRNet, the strongest reported numbers are the original MRNet AUCs of 0.965 for ACL tear, 0.937 for abnormality, and 0.847 for meniscal tear MRNet.
Beating these baselines means three things. First, mean AUROC stays within a pre-registered margin of 0.01 of the matched fine-tuned baseline. Second, deletion-curve faithfulness and SynVol rationale F1 are clearly better. Third, the borrowed fraction predicts accuracy loss on partner-absent subsets better than any signal we can extract from the baselines' explanations.
Three ablations isolate the key ingredient. The first is a 3D Soft-CAM head with independent channels and no borrowed term. It asks whether explicit transfer matters at all. The second replaces $A_{kj}(x,v)$ with one learned global matrix, $A_{kj}$ constant across scans and locations, in the spirit of ML-GCN. It asks whether local conditioning matters. The third removes the balanced-partner loss. It asks whether the direct channel stays clean without it. A fourth ablation runs the same head on slice tokens taken before the inter-slice transformer, with max pooling over depth. This measures how much volumetric context contributes to evidence continuity across slices.
Why not just…?
Why not just use a strong 3D model with better post-hoc attribution? Post-hoc methods run backward from an output score. In a multi-label model the gradient of label $k$ flows into whatever features predict $k$, including features belonging to a correlated finding. The heatmap therefore shows the partner's location with no marker that the evidence was borrowed. MST's 0/150 Grad-CAM result and Soft-CAM's deletion-curve comparisons already show poor fidelity in single-label settings. SynVol tests the multi-label case directly. If Integrated Gradients on fine-tuned CT-CLIP recovers the inferred-label routing as well as our head does, the architectural contribution fails, and we will report that.
Why not just extend Soft-CAM to 3D with independent class channels? This is the most serious threat. It is our first ablation, and it may match our accuracy. We expect it to fail in a specific way. With no explicit route for correlation, the only place a correlation can show up is inside the evidence map of the correlated label. On SynVol, the inferred label's map would then sit on finding A's lesion and could not be distinguished from direct sight. If the independent head's maps for correlation-only labels turn out empty, and its accuracy holds, then correlation is not being exploited in that regime and our borrowed channel is unnecessary there. The decisive check is where each head places the inferred label's evidence.
Why not just use a static global correlation matrix? A global matrix transfers evidence from A to B in every scan, including scans where the relationship does not hold. That is the noise Huang et al. describe. Chochlakis et al. found that per-example constraints beat static ones. Neither result concerned explanations or 3D data, so the global-matrix ablation settles the question here. The prediction is that the global variant over-borrows on SynVol's inverted-correlation test set and loses accuracy there, while the local variant does not.
Why not just balance the training data and skip the decomposition? Reweighting alone can reduce reliance on co-occurrence. But it also discards useful inference: a finding with no visual signature, such as MRNet's "abnormal" when implied by a tear, cannot be predicted without borrowing. The third ablation and a full-model variant that uses only reweighting measure this trade-off directly.
Decisive Experiments
On SynVol, we score direct-map localization against planted lesion masks with top-k precision. We score channel routing by requiring $\rho_k$ to be near 1 for the correlation-only label and near 0 for independent labels. We also run a counterfactual test: remove finding A's planted lesion and measure the change in $s_B$. In our model the head's share of that change is computable exactly. The backbone's share is not exact, because transformer context mixes tokens, so we report measured agreement and do not claim exactness.
On MRNet, the natural test is the "abnormal" label, which is largely implied by ACL and meniscal tears. The model should assign it a high borrowed fraction whose borrowed evidence sits on the tear, plus direct evidence in tear-free abnormal cases.
On CT-RATE, we report mean AUROC against the fine-tuned CT-CLIP and MST baselines. We report deletion-curve area for every explainer. We report the rank correlation between $\rho_k$ and the per-finding AUROC drop on partner-absent subsets. Finally, a radiologist rates 150 volumes, blinded as in MST, for whether direct evidence falls on the named finding. This keeps the fidelity versus plausibility tension in view and does not assume it away.
The project succeeds if the independent-channel and global-matrix ablations both fail on routing or on shift prediction while our head holds accuracy. It fails honestly if a simpler variant matches us on all three measures.
References
- Multi-Label Learning by Exploiting Label Correlations Locally
- Medical slice transformer for improved diagnosis and explainability on 3D medical images with DINOv2
- Soft-CAM: Making black box models self-explainable for medical image analysis
- Conditional Graphical Lasso for Multi-label Image Classification
- HOT-VAE: Learning High-Order Label Correlation for Multi-Label Classification via Attention-Based Variational Autoencoders
- Self-Explainable Multi-Label Graph Neural Network for Correlated Evidence Attribution
- A foundation model utilizing chest CT volumes and radiology reports for supervised-level zero-shot detection of abnormalities
- Deep-learning-assisted diagnosis for knee magnetic resonance imaging: Development and retrospective validation of MRNet
- Leveraging label correlations in a multi-label setting: A case study in emotion