When a medical imaging model moves to a new hospital, two things break at once. The scans look different because the scanner, slice thickness, or reconstruction settings differ. And the cases that matter most, such as highly suspicious nodules or rare lesion types, are the ones the new site has too few labeled examples of to retrain on. We propose to use the structure of a concept bottleneck model to handle both problems. A concept bottleneck model first predicts human-readable findings (for a lung nodule: spiculation, lobulation, margin sharpness, texture, calcification) and then predicts the diagnosis from those findings with a simple, transparent layer. Our claim is that acquisition shift damages a small number of these finding detectors in a predictable way. Because findings are shared across diagnoses, a few dozen labeled common cases from the new site are enough to recalibrate the damaged detectors, and this repair carries over to rare diagnoses the new site has barely seen.
In two sentences: protocol shift corrupts specific intermediate findings rather than the whole model, and since rare and common diseases share findings, fixing those findings with cheap, plentiful common-case labels should recover rare-case sensitivity. A black-box classifier offers no such shared, inspectable intermediate, so it has to see rare cases from the new site to adapt to them. The project tests this, and also measures how much a black box actually needs those rare cases.
Core Claim
We make three linked, falsifiable claims. First, under real acquisition variation, concept prediction errors concentrate in a few concepts, and which concepts fail follows the physics of the shift. Thick slices should hurt fine-margin concepts such as spiculation. Smoothing reconstruction kernels should hurt texture. Second, in concept bottleneck models that add a free "unknown concept" channel to close the accuracy gap, the rare-class predictions rely more heavily on that channel than common-class predictions do, and that channel carries acquisition-site information. This combination is what makes rare-class sensitivity collapse across sites. Third, a per-site recalibration of only the drifted concept heads, fitted on a small labeled sample drawn mostly from common classes, recovers rare-class sensitivity at least as well as fine-tuning a class-rebalanced black box on the same labels.
Where This Sits in Prior Work
The motivation comes from work showing that concept bottlenecks resist confounding. The original formulation by Koh et al. defines the two-stage image-to-concepts-to-label model we build on. Robust and Interpretable Medical Image Classifiers via Concept Bottleneck Models projects chest X-rays onto GPT-4-generated descriptors through a medical vision-language model. On the NIH-age confounded split it reports 50.7% accuracy versus 9.4% for raw features. KnoBo adds textbook-derived priors on the concept-to-label weights and reports large gains over fine-tuned Vision Transformers under site and demographic confounding. Both papers study 2D images, spurious correlations with labels, and balanced evaluation. Neither asks how individual concept detectors degrade under physical acquisition change, and neither studies rare classes. We take their robustness result as the reason to expect concepts to be a useful place to intervene. We do not take it as evidence that concepts are invariant.
A second group supplies the tension we resolve. CCBM adds concept adapters and an "unknown concept" branch that comes within 1–2% of black-box accuracy on Derm7pt, with 93.96% concept-detection AUC. The same paper reports unstable results on small or regression-style concept sets such as BrEaST and LIDC-IDRI. Liu et al. show that saliency maps are associative. They force decisions onto expert lesion regions through counterfactual training, reaching 72.2% accuracy and 75.1% localization precision on LIDC-IDRI with shortcut artifacts, but at the cost of second-order optimization. CCBM's free channel is exactly the kind of unconstrained path Liu et al. and KnoBo warn can re-learn shortcuts. We test that concern directly by measuring site information in the channel and constraining it cheaply, instead of adopting Liu et al.'s expensive counterfactual training.
Three papers set constraints on the design. PN-SAMP shows that jointly predicting nodule attributes and malignancy on LIDC-IDRI is accurate (97.58% malignancy accuracy, 89.33% attribute accuracy). It shows no cross-acquisition evaluation, so we reuse its attribute set and treat its within-dataset number as an upper reference, not a cross-site one. Red-GAN shows that imbalance and acquisition center interact: conditioning synthesis on center improves BraTS by 5%. The same paper names the "synthesis dilemma", that generators need the data rare classes lack, which is why we avoid generation for the rare classes. The screening mammography meta-repository and the Giammanco et al. meta-analysis (pooled heterogeneity $I^2 = 84.72\%$) establish the reproducibility standard we adopt. That standard means containerized head-to-head evaluation on shared cohorts, with precision-recall metrics suited to low prevalence.
Research Program
The first component is a concept-level drift audit. We train concept bottleneck models on LIDC-IDRI's radiologist-scored nodule attributes. We then evaluate each concept head separately on held-out acquisition strata, defined by slice thickness, reconstruction kernel, and scanner manufacturer from the DICOM headers. We also evaluate on controlled perturbations of held-in scans: slices resampled from thin to 2.5 mm and 5 mm, and smoothing that mimics softer kernels, in the spirit of the domain randomization in SynthSR. Per concept, we record discrimination and calibration change as a dose-response curve over perturbation strength. The prediction to test is that drift is sparse and matches the physics of the shift. If drift is instead diffuse across all concepts, the repair idea loses its footing, and we would report that.
The second component is a channel attribution analysis for models with an unknown-concept branch. We measure how much of each class's logit comes from named concepts versus the free channel, stratified by class frequency. We then train a linear probe to predict acquisition stratum from the free channel alone. We then constrain the free channel to carry no acquisition information given the label. This uses a conditional adversarial penalty on that channel only. It is much cheaper than the implicit-function counterfactual training of Liu et al. because it needs no Hessian products.
The third component is the repair itself. For each target site $s$ and each drifted concept $k$, we refit a two-parameter correction of the concept logit,
$$\tilde{c}_k^{(s)} = \sigma\!\left(a_k^{(s)}\, z_k + b_k^{(s)}\right),$$
where $z_k$ is the source model's logit for concept $k$, $\sigma$ is the logistic function, and $a_k^{(s)}, b_k^{(s)}$ are fitted on $n$ labeled nodules from site $s$. For ordinal attributes the same correction is applied to the thresholds between levels. The concept-to-label layer stays frozen, so the clinical reasoning a radiologist can inspect is unchanged. We fit only drifted concepts, as flagged by the audit using unlabeled target data through the shift in predicted concept distributions, and draw the $n$ cases from whatever the site has, which will be mostly benign or low-suspicion nodules.
The fourth component is the reproducibility package. It includes fixed acquisition-stratified splits, the perturbation code, and every model and baseline, packaged in containers following the meta-repository's approach so that any group can rerun the head-to-head comparison.
Baselines
The naive reference is a black-box 3D convolutional network trained on source strata with standard cross-entropy and applied to target strata with no adaptation. It shows how large the cross-acquisition drop is in the first place. No reported result was found on Lacuna for LIDC-IDRI under acquisition-held-out splits. The closest reported figure is PN-SAMP's within-dataset 97.58% malignancy accuracy (Wu et al.), which we reproduce on our in-distribution split as a sanity check before any cross-site comparison.
The strongest competitor on the adaptation question is the same black box fine-tuned on the identical $n$ target-site labels. It is trained with a class-rebalanced objective (logit adjustment for class frequency) and its last layers are unfrozen. It is the strongest competitor because it uses the same labels, has more adjustable capacity, and directly addresses imbalance. It gets the same target samples, the same random seeds, and a hyperparameter search budget equal to or larger than ours. We count it as beaten only if our repair gives higher rare-class sensitivity at 90% specificity and higher precision-recall AUC, with non-overlapping bootstrap intervals across at least three target strata and at $n \in \{25, 50, 100\}$.
The strongest interpretable competitor is CCBM, trained on the same split and evaluated with and without target fine-tuning. On Derm7pt it reports 93.96% concept-detection AUC and diagnosis within 1–2% of black-box models. It reports no single overall LIDC-IDRI number. A pure concept bottleneck without the free channel, following Koh et al., tests whether removing the channel is enough. We also include acquisition harmonization: resampling all scans to common spacing, with kernel-matching smoothing, before a black box.
Three ablations isolate the key ingredients. The first recalibrates all concept heads instead of only the flagged ones, to test whether the audit matters. The second omits the free-channel constraint. The third fits the repair on target labels that exclude every rare-class case. This last one directly tests whether findings shared with common cases carry rare-class repair.
Why not just…?
Why not just fine-tune the black box on the new site's labels with a class-balanced loss? This is the real threat, and the proposal is designed to settle it. With few target labels and nearly no rare cases among them, a black box has no shared, named intermediate that ties common-case corrections to the rare-class decision region. Rebalancing cannot upweight rare cases the target sample does not contain. That is the argument, not evidence. The rare-excluded experiment decides it. If the fine-tuned black box matches our repair when neither sees target rare cases, then the transfer-through-shared-concepts claim is wrong. We would reframe the contribution as the drift audit and the interpretability of the fix.
Why not just harmonize the images before classification? Pixel-level harmonization makes images look alike but gives no guarantee that fine clinical detail survives. Shakya & Khanal show high overlap scores coexisting with failed anatomical measurements, and a 30% Dice drop under a 10-degree tilt. Resampling thick slices cannot restore spiculation that the acquisition never captured. The harmonization baseline and the dose-response curves will show whether it recovers the specific concepts that drift.
Why not just drop the unknown-concept channel? Doing so removes the leakage path, but CCBM's results suggest named concepts alone leave an accuracy gap. The pure-bottleneck baseline measures that gap on LIDC-IDRI. If it is small and the pure model degrades no worse across sites, the free-channel constraint is unnecessary and we will say so.
Why not just use domain-adversarial training on the whole network? Whole-network invariance to site pushes against concept detectors that legitimately change with acquisition. It also offers no per-concept account of what failed. Running it within the same budget will show whether it beats targeted recalibration on rare-class sensitivity.
Decisive Checks
The primary benchmark is LIDC-IDRI with acquisition-stratified held-out splits and controlled resampling perturbations. The rare classes are the highest-suspicion malignancy scores and the uncommon calcification patterns. Metrics are precision-recall AUC, sensitivity at 90% specificity for the rare classes, and per-concept AUC and calibration error. We replicate on CBIS-DDSM, using its mass shape and margin descriptors as concepts and digitizer type, where recorded, as the acquisition stratum. This lets us check the claim against the causal-alignment numbers reported in the Liu et al. repository, where baseline DDSM accuracies range from 0.559 to 0.592.
Three results would each count against the idea: diffuse rather than sparse concept drift; no excess reliance of rare classes on the free channel; or a rare-excluded black-box fine-tune that matches concept repair. The idea is supported if drift is sparse and physically predictable and if repair fitted on common-case labels recovers rare-class sensitivity where the black box does not. That outcome would give hospitals a cheap, auditable way to adapt diagnostic models to their scanners.
References
- Concept Bottleneck Models
- Robust and Interpretable Medical Image Classifiers via Concept Bottleneck Models
- A Textbook Remedy for Domain Shifts: Knowledge Priors for Medical Image Analysis
- Concept Complement Bottleneck Model for Interpretable Medical Image Diagnosis
- Learning Causal Alignment for Reliable Disease Diagnosis
- Joint learning for pulmonary nodule segmentation, attributes and malignancy prediction
- Red-GAN: Attacking class imbalance via conditioned generation. Yet another medical imaging perspective
- Meta-repository of screening mammography classifiers
- Diagnostic Accuracy of Artificial Intelligence for Detection of Meniscus Pathology on Magnetic Resonance Imaging: A Systematic Review
- SynthSR: A public AI tool to turn heterogeneous clinical brain scans into high-resolution T1-weighted images for 3D morphometry
- Benchmarking Encoder-Decoder Architectures for Biplanar X-ray to 3D Bone Shape Reconstruction