When a deep network reads a 3D CT scan and says "malignant," the usual way to explain it is a heatmap: a saliency method that colors the voxels the network seemed to rely on. In three dimensions these heatmaps fail badly. The SE3D benchmark found that Grad-CAM and HiResCAM localize nothing on a brain MRI task, scoring 0.00 Max3DBoxAcc (SE3D). We propose to drop the heatmap and make the explanation part of the model. The network first draws a 3D outline (a mask) of the lesion. Fixed, differentiable formulas then measure a handful of physical quantities from that outline and the CT intensities inside it, such as volume, sphericity, elongation, boundary contrast, and density spread. A small piecewise-linear function borrowed from materials science turns those measurements into the diagnosis.
Each prediction can then be checked in three places: the outline (does it cover the nodule?), the measurements (what is this nodule's sphericity?), and the active linear piece (which of a few learned "phenotype families" it falls in, and with what weights). The model cannot draw on image content outside the outline and its boundary band except through the outline's shape. The open question is how much accuracy this costs compared with black-box 3D networks and with existing concept-bottleneck models on lung nodules. We will measure that cost directly.
Core Claim
A concept bottleneck (a network forced to predict human-meaningful attributes before its final answer, as in Concept Bottleneck Models) usually predicts its concepts with learned heads. That design has two known weaknesses. The predicted "concept" can quietly carry information unrelated to its meaning, which is concept leakage. And in 3D the concepts are typically subjective radiologist scores, as in PN-SAMP (Joint Learning for Pulmonary Nodule Segmentation, Attributes and Malignancy Prediction), which jointly segments nodules, regresses nine radiologist-scored attributes, and predicts malignancy. Its attribute heads and its malignancy head both read encoder features, so the attributes do not actually bottleneck the decision.
We claim that computing concepts as deterministic geometric and intensity integrals over a predicted 3D mask removes the leakage channel by construction. The mask is a visual, checkable object, and a measurement such as sphericity cannot secretly encode texture elsewhere in the scan. The decision head is a max-affine function, the same form used in A seven-facet polyhedron captures the composition-only formation-energy landscape of inorganic solids. That paper describes formation energies of inorganic solids as the maximum of seven linear functions and reads the facets as chemical families. Here the head is
$$ z(c) = \max_{f=1..K} \left( w_f^\top c + b_f \right), $$
where $c$ is the vector of measured concepts, $K$ is a small number of facets, $w_f$ and $b_f$ are the weights and offset of facet $f$, and $z$ is the malignancy logit. Each case receives a facet assignment, which forms a discrete phenotype family, and an exact linear decomposition within it.
The materials literature supplies the method and a warning. On formation energies the seven-facet model reached 0.076 eV/atom error against 0.044 eV/atom for a shallow neural network (polyhedron paper). Glass-box heads cost accuracy, so the cost has to be measured rather than assumed away.
Research Program
The segmenter is a compact 3D U-Net, following 3D U-Net and PN-SAMP's reduced-parameter variant. It runs on nodule-centered LIDC-IDRI patches with the same multi-window CT inputs PN-SAMP uses. Its soft mask $m$ feeds a fixed concept layer of about eight to twelve measurements. Volume is $\sum m$. Surface area comes from the summed magnitude of the mask gradient. Sphericity is computed from volume and surface area. Elongation and flatness come from the eigenvalues of the mask's second-moment matrix. Mean and spread of intensity are mask-weighted statistics in Hounsfield units. Boundary contrast is the intensity difference across a thin band at the mask surface. Margin irregularity is the variance of the radial distance from the centroid to the surface. Concepts are standardized using training-set statistics so the facet weights are comparable.
The intensity measurements read the image only through $m$ and a band around it. This gives a hard locality guarantee for intensity evidence. Shape evidence still depends on context, because the segmenter sees the whole patch, and we say so plainly.
Training proceeds in two stages. The segmenter is first trained for Dice on the expert masks. It is then fine-tuned end-to-end with the malignancy loss plus a retained segmentation loss. We track Dice throughout, because a mask that drifts away from the anatomy to improve classification would make the explanation dishonest. We also train $K$ from 1 to 8 to trace the accuracy-against-simplicity curve. $K=1$ is a linear head.
The analysis has three parts. First, we test whether the learned facets correspond to recognizable nodule families. We do this by comparing facet membership against the radiologist attribute labels that the model never saw. Second, we test whether the measured concepts track those labels; for example, the irregularity measure should rank-correlate with radiologist spiculation scores. Third, we compute how far the mask-based evidence region and Grad-CAM on a black-box 3D CNN each fall from the expert nodule mask, using SE3D's voxel metrics.
Baselines
The naive reference point predicts the majority class and also fits logistic regression on nodule diameter alone. Diameter is the single measurement radiologists already use. Any concept model that does not clearly beat diameter has added no useful information.
The strongest existing method is PN-SAMP (Joint Learning for Pulmonary Nodule Segmentation, Attributes and Malignancy Prediction). It reports 97.58% malignancy accuracy, 73.89% Dice, and 89.33% attribute accuracy on LIDC-IDRI, against 86.84% for MCNN and 91.26% for CNN-MTL. It is the closest competitor because it already combines 3D segmentation, clinical attributes, and malignancy on the same data. We will rerun it under our patient-level splits with the same input windows, epoch budget, and hyperparameter search as our model. If our reproduction falls short of 97.58%, we report both numbers and compare against the reproduced figure.
The comparison also includes an unconstrained 3D ResNet trained only on malignancy labels, which serves as the accuracy ceiling for an opaque model, with Grad-CAM supplying its explanation. It also includes a standard concept bottleneck that predicts the nine radiologist attributes from encoder features and then applies a linear head. That model isolates "measured from the mask" versus "predicted from features."
The ablation that isolates our key ingredient swaps measured concepts for an equal number of free learned scalars computed from mask-pooled features, keeping the same max-affine head. If this gains accuracy, that gain quantifies the hidden information our design refuses to use. A second ablation compares $K=1$ with the best $K$, isolating the value of the facet structure.
We define winning in two parts. On accuracy, the measured-concept model must come within two points of the stronger of reproduced PN-SAMP and the 3D ResNet, and clearly beat the diameter baseline. On explanation, its evidence region must score far above Grad-CAM on SE3D voxel metrics against expert masks, while its concepts correlate with held-out radiologist attributes at least as well as the predicted-concept bottleneck's.
Why not just…?
Why not just run 3D Grad-CAM on a strong black-box 3D CNN? That is cheaper and keeps full accuracy. But the evidence says the explanations are not trustworthy in 3D. SE3D reports best-case Max3DBoxAcc of 0.19 to 0.63 across 3D tasks and 0.00 for Grad-CAM and HiResCAM on brain MRI (SE3D). We do not take this on faith for nodules. Our localization experiment scores Grad-CAM on LIDC-IDRI with the same metrics. If Grad-CAM localizes nodules well on small pre-cropped patches, which is plausible since the nodule fills much of the patch, the localization argument weakens. The claim would then rest on leakage-free concepts and facet structure.
Why not just compute classical radiomics shape and intensity features from a frozen segmenter and fit logistic regression or boosted trees? This is the most serious threat, because our concept layer is close to a differentiable radiomics set. We include it directly. The same concept formulas are computed from the frozen stage-one masks and fed to logistic regression, gradient-boosted trees, and the max-affine head. If the frozen pipeline with a max-affine head matches the end-to-end model, end-to-end fine-tuning has added nothing. The contribution then shrinks to the max-affine facet head and the evaluation protocol, and we will report it that way. Boosted trees will likely be somewhat more accurate but lose the facet reading, which is the trade-off the $K$ sweep makes visible.
Why not just use a standard concept bottleneck on the radiologist attributes, as PN-SAMP's attribute heads already allow? Those attributes are subjective consensus scores, and the file notes this as a limitation of PN-SAMP. A learned concept head can also carry information beyond its label, as the concept-leakage concern in the research file describes. The free-scalar ablation and the attribute-bottleneck baseline measure both effects. If predicted radiologist concepts give equal accuracy and equal correlation with held-out attributes, the case for measured concepts rests only on the locality guarantee.
Why not a linear head or a spline-based additive model instead of max-affine? A linear head is the $K=1$ ablation. An additive model with one B-spline curve per concept, in the spirit of Ultra-fast interpretable machine-learning potentials, is a reasonable alternative glass-box head, and we run it with the same budget. Max-affine earns its place only if its facets form coherent nodule families. If the additive model is as accurate and its curves are easier to read, we will recommend it instead.
Proposed Experiments
The primary benchmark is LIDC-IDRI, with 1,010 patients and 2,660 nodules carrying 3D masks, nine attributes, and malignancy ratings. We use five-fold patient-level cross-validation. We report malignancy accuracy and AUC, Dice before and after end-to-end fine-tuning, and Spearman correlation between each measured concept and its closest radiologist attribute. Attributes are never used in training our model.
The first decisive check is the accuracy cost curve over $K$ against the 3D ResNet and reproduced PN-SAMP. A gap larger than about two points at every $K$ would mean measured concepts are too incomplete a vocabulary for this task. We would then report the gap honestly rather than adding free scalars back in.
The second check scores the evidence regions of all models against expert nodule masks with SE3D's MaxF1, voxel average precision, and mass concentration.
As a secondary test of the locality guarantee, we train the same model on SE3D's brats-halves task, using BraTS tumor masks to supervise the segmenter. We compare its evidence region's Max3DBoxAcc with the reported 0.00 for Grad-CAM. The task is easy for a mask-based model, so we treat it as a sanity check, not a headline result.
The third check is a perturbation test. We alter intensities outside the mask band and confirm that the logit changes only through changes in the mask, while the 3D ResNet's logit changes freely.
Finally, a facet-coherence test asks whether facet membership predicts held-out radiologist attributes better than k-means clustering of the same concepts. A negative result would mean the facets are an optimization artifact rather than phenotypes.
References
- SE3D: A Framework for Saliency Method Evaluation in 3D Imaging
- Concept Bottleneck Models
- Joint learning for pulmonary nodule segmentation, attributes and malignancy prediction
- A seven-facet polyhedron captures the composition-only formation-energy landscape of inorganic solids
- 3D U-Net: Learning Dense Volumetric Segmentation from Sparse Annotation
- Ultra-fast interpretable machine-learning potentials