When a medical segmentation model moves to a new hospital or scanner, it does not only get worse. It gets worse while staying confident, and that combination is what makes it unsafe. On the AMOS abdominal benchmark, models lose more than 20 points of mean Dice on MRI scanners they never saw in training. The most common outputs in that situation are organs placed in implausible positions: liver tissue predicted in the lower pelvis, a kidney split across the midline, an adrenal gland ten times its usual size. A clinician cannot tell these failures apart from correct output, because the model's per-voxel probabilities look the same in both cases. We propose a simple check. Divide the body into a pyramid of blocks: the whole scan, then halves along each axis, then quarters. In each block, record which organs the model claims are present and in what proportions. Then ask how probable that layout is under the layouts seen in the training labels. Blocks whose predicted composition is implausible at some level of the pyramid receive a higher temperature, meaning their probabilities are softened. Their sum also gives a scan-level warning score.
The two-sentence version is this. Spatial pyramids, the classic trick of summarizing an image block by block at several grid sizes so that "where" is kept alongside "what", can be applied to a model's predicted label map rather than to the image. The result is a layout prior that knows nothing about anatomy beyond the training labels, needs no target-hospital data, and turns implausible regional organ composition into calibrated uncertainty under domain shift. The key difference from existing shape-prior calibration is that the prior models label counts per block, not organ contours. It should therefore work for many organs at once and for organs with variable shape, which is where whole-shape priors break.
Core Claim
Predicted label composition, measured at several nested spatial granularities, is a stronger and more general signal of out-of-domain failure than either voxel-level perturbation sensitivity or a learned whole-image shape prior. We expect the multi-level version to beat both single-level versions. The global level alone (organ volume fractions) misses misplacements that preserve volume. A flat fine grid alone is too noisy and sparse to estimate. The intermediate levels carry the "this organ is in the wrong region" evidence. The claim is testable: on held-out MRI scanners in AMOS, a temperature map driven by the layout pyramid should reduce Expected Calibration Error (the gap between confidence and accuracy, averaged over confidence bins) and improve scan-level failure detection, compared with a matched single-level ablation and with the current shape-prior calibration method.
Relation to Prior Work
The method we extend directly is Improved post-hoc probability calibration for out-of-domain MRI segmentation. It combines each pixel's sensitivity to test-time perturbation with a denoising-autoencoder shape prior to produce a spatially varying temperature map. That method lowered ECE from 16.78% to 15.37% on cardiac bSSFP→LGE. Its prior, however, reconstructs whole shapes and assumes regular morphology such as cardiac ventricles. We keep its overall structure: a post-hoc, spatially varying temperature fitted only on source data. We replace the shape autoencoder with a count model over pyramid blocks, which needs no organ to be compact.
The pyramid itself comes from classical spatial pyramid matching. Oxford/IIIT TRECVID 2008 aggregates descriptors inside recursively partitioned regions, and An Extendible Hash for Multi-Precision Similarity Querying of Image Databases uses sub-block histograms to separate images whose global statistics coincide but whose layouts differ. Both work on image appearance. We apply the same construction to semantic label maps, where the layout is far more stable across hospitals than raw MRI intensities are. To score the per-block organ counts, we reuse the nested Dirichlet count model of A hierarchical count data clustering based on Multinomial Nested Dirichlet Mixture, which fits proportion vectors with hierarchical structure better than a flat multinomial.
Existing work joining spatial hierarchies with domain shift does something different in each case. Spatial Attention Pyramid Network for Unsupervised Domain Adaptation pools features over a pyramid and aligns them adversarially, so it needs unlabeled target images, and it was tested only on driving scenes. HASD regularizes pathology slides at domain, slide and patch levels during adaptation, again using target data. FundusAdapter uses a fixed two-tier global/local patch split to adapt a fundus foundation model. StyCona and MADGNet decompose images algebraically (singular values) or spectrally (frequency bands) to train more robust segmenters. All five change features or training. None of them uses a spatial hierarchy over outputs to decide when to distrust a model. Our proposal is purely post hoc and never sees target data. These papers motivate our view that multi-level structure matters under shift, but they are not competitors for the calibration question.
Two findings constrain the design. AMOS (AMOS) reports that a plain 3D U-Net (89.04% mDice on CT) beats Swin-UNETR (86.32%). We therefore fix a strong convolutional segmenter and do not claim architectural gains. AMOS also documents the sharpest degradation on small structures such as the adrenal glands and duodenum. This means calibration has to be reported per organ, because averages over large organs hide failures on small ones.
Research Program
The first component is a canonical pyramid. Abdominal scans differ in field of view, so the grid is placed on the body bounding box. This box is found from a foreground mask by intensity thresholding and morphology, which works the same way on CT and MRI. The box is split into levels $l = 0, 1, 2, 3$, with $1$, $2^3$, $4^3$ and, as an option, $8^3$ blocks. For each block $b$ we compute the soft label-proportion vector: the predicted softmax probabilities averaged over the block's voxels, with background included. On source training labels, each level gets its own nested Dirichlet mixture over block proportions, conditioned on block position. A block in the upper-right abdomen thus has its own expected composition.
The second component is the temperature map. For voxel $i$, let $b_l(i)$ be the block containing it at level $l$. Let $s_l(b)$ be the negative log-likelihood of block $b$'s predicted composition under the level-$l$ model, standardized by its mean and spread on held-out source scans. The temperature is
$$T_i = \exp\Big(a + c\,u_i + \sum_{l} w_l\, s_l(b_l(i))\Big),$$
where $u_i$ is the voxel's perturbation sensitivity as defined in the shape-prior calibration paper, and $a$, $c$ and $w_l$ are scalars. They are fitted by minimizing negative log-likelihood on source validation scans that have been degraded with synthetic MRI artifacts, following that paper's source-only protocol. No target scans are used to fit any parameter. The scan-level failure score is the volume-weighted sum of the $s_l$ terms.
The third component is analysis. We will measure which levels carry the signal, learning the weights $w_l$ and also zeroing them one level at a time. We will check whether the signal concentrates in blocks that border small organs. We will also look for an inverse failure: cases where a correct but unusual anatomy, such as situs variants, post-surgical absence of an organ, or large tumors, is flagged as implausible. Such false alarms are the main clinical cost of any layout prior and will be reported, not filtered out.
Baselines
The naive reference point is the uncalibrated softmax of the segmenter. Next to it is ordinary global temperature scaling fitted on source validation data. Both are evaluated on the same held-out target scans, and this is the level any method must clear.
The strongest existing method is the spatially varying calibration of Improved post-hoc probability calibration for out-of-domain MRI segmentation, including its local temperature scaling (LTS) baseline. We will re-implement it on the identical frozen segmenter. Its denoising autoencoder will be trained on the same source label maps that we use for our count models. Its perturbation set and synthetic-artifact validation will be shared with ours, so the only difference between the methods is the prior. On MS-CMR bSSFP→LGE, that paper reports ECE of 15.37%, against 16.78% for LTS. On simulated artifacts it reports 12.78%, against 14.48%. Beating it means lower ECE than our reproduction of it on the same split, and reproducing its reported number within a reasonable tolerance before claiming anything. On AMOS, no reported calibration result was found on Lacuna. There we compare only against our own reproduction, and we require a per-organ improvement on the small organs, not just an improvement in the mean.
Deep ensembles of five independently trained segmenters serve as a strong generic uncertainty baseline. They get five times the training compute, and we disclose this. We also test our temperature map applied on top of the ensemble, since the practical question is whether the layout signal adds information an ensemble lacks.
The ablation that isolates the key ingredient keeps everything fixed and varies only the pyramid. One variant uses level 0 alone, meaning global organ volume fractions. Another uses the finest level alone, a flat grid. Both are compared with the full multi-level pyramid. A further control replaces the position-conditioned Dirichlet mixture with a single unconditioned multinomial per level, to test whether the count model matters.
Why not just…?
Why not just use a deep ensemble? Ensembles do improve calibration, and under shift they are a real threat. However, ensemble members trained on the same source data tend to make the same layout mistakes on an unseen scanner, because they share the same intensity cues. Where they agree confidently on a wrong placement, the ensemble's uncertainty stays low. The test is direct: we stratify target scans by whether the ensemble's members agree, and check whether the layout score still detects failures among high-agreement scans. If the ensemble alone matches ensemble plus pyramid on ECE and failure-detection AUROC, the proposal's practical value is limited to cases without the compute for an ensemble, and we will say so.
Why not just remove the shift with intensity augmentation, such as random bias fields, contrast jitter, or StyCona-style mixing? Augmentation reduces the shift but does not report the leftover failures. StyCona (StyCona) still reaches only 73.39% Dice on bSSFP→LGE, so many errors remain. Calibration and robustness are complementary. We will train the segmenter with and without strong augmentation, including StyCona-style mixing, and calibrate both. The claim survives only if the layout pyramid still improves calibration on the augmented model.
Why not just use a whole-shape prior, which already exists? The denoising autoencoder prior is designed for regular shapes, and its authors note that it struggles with irregular morphology. With fifteen abdominal organs, whole-volume reconstruction must model the joint shape of all organs at once. Block-level composition only asks which organs are present in each region and in what amounts. On AMOS's small organs and on scans with tumors, this should degrade more gracefully. The head-to-head comparison on identical inputs will settle the question, and if the autoencoder wins on MS-CMR's compact cardiac anatomy, that result is expected and will be reported.
Why not just use global organ volumes, without a pyramid? This is the level-0 ablation. A model that places the right amount of liver in the wrong place passes a volume check. Intermediate levels are where misplacement shows up, consistent with the multi-resolution results in Extending scene-to-patch models, where predicting at several patch scales beat a single resolution (0.417 vs 0.397 mIoU on DeepGlobe). If level 0 alone matches the full pyramid, the hierarchical part of the claim is false.
Proposed Experiments and Decisive Checks
The main experiment uses AMOS. A 3D U-Net is trained on CT plus MRI from a subset of scanners, and tested on MRI from the held-out scanners. The organs scored are those annotated in both training and test modalities. A second split goes from CT to MRI with no MRI in training. We report voxel-level ECE overall and per organ, Brier score, scan-level failure-detection AUROC (where failure means scan Dice below a fixed threshold), and the area under the risk-coverage curve for deferring scans to a radiologist. The external anchor is MS-CMR bSSFP→LGE and its simulated-artifact variant, where reproducing the reported 15.37% and 12.78% ECE is the precondition for every comparison. PolypGen is used as a deliberate negative control: polyp position is close to arbitrary, so the layout pyramid should add little there. A large gain on PolypGen would point to a leak or artifact rather than a layout effect.
Three outcomes decide the idea. First, the full pyramid must beat the level-0 and finest-level ablations on held-out-scanner ECE and failure-detection AUROC in AMOS. Second, it must beat the reproduced shape-prior method on AMOS small organs while not losing badly on MS-CMR. Third, it must add value on top of a five-member ensemble. If the first fails, hierarchy is not the active ingredient. If the third fails, the method is a cheap substitute for ensembling rather than a new source of information, which is still useful but a narrower contribution. Throughout, we track false alarms on anatomically atypical but correctly segmented scans, because a layout prior that flags every unusual patient is not safe to deploy.
References
- Improved Post-hoc Probability Calibration for Out-of-Domain MRI Segmentation
- Oxford/IIIT TRECVID 2008 - Notebook paper
- An Extendible Hash for Multi-Precision Similarity Querying of Image Databases
- A hierarchical count data clustering based on Multinomial Nested Dirichlet Mixture using the Minorization-Maximization framework
- Spatial Attention Pyramid Network for Unsupervised Domain Adaptation
- HASD: Hierarchical Adaption for Pathology Slide-Level Domain-Shift
- FundusAdapter: few-shot adaptation of fundus image foundation model for fundus image diagnosis
- Style Content Decomposition-based Data Augmentation for Domain Generalizable Medical Image Segmentation
- Modality-Agnostic Domain Generalizable Medical Image Segmentation by Multi-Frequency in Multi-Scale Attention
- AMOS: A Large-Scale Abdominal Multi-Organ Benchmark for Versatile Medical Image Segmentation
- Extending scene-to-patch models: Multi-resolution multiple instance learning for Earth observation