A U-Net is the standard network for outlining organs and tumours in medical images. It compresses the image through an encoder, rebuilds a full-resolution mask through a decoder, and passes shortcuts called skip connections from each encoder stage directly to the matching decoder stage. Those shortcuts are why U-Nets draw sharp boundaries. They may also be why U-Nets break at a new hospital. The high-resolution shortcuts carry fine texture straight to the output without passing through the compressed middle of the network, and fine texture is where scanner-specific noise, reconstruction kernels, and contrast signatures live. This proposal first measures how much of a U-Net's cross-hospital failure travels through each skip connection when the network is trained on fewer than about fifty cases from a single center. It then tests a targeted fix: train only the offending shortcuts to ignore simulated scanner changes, and leave the rest of the network free.
The work also matters downstream of segmentation. Many clinical prediction models are still built from handcrafted radiomic features, meaning predefined measurements of shape, intensity histograms, and texture computed inside a tumour mask. These features are known to be fragile to small boundary errors. If automated masks drift at an external site, every downstream radiomic model inherits that drift. We therefore judge the fix by segmentation overlap and also by whether the radiomic features computed from automated masks stay stable across sites.
Core Claim
Under small, single-center training, the main route by which acquisition differences corrupt U-Net segmentations is the one or two highest-resolution skip connections, not the bottleneck. The claim has three testable parts. First, a test-time attribution procedure can localize the damage to specific skip levels. Second, a consistency penalty applied only at those skip levels, using a family of simulated acquisition changes, recovers more external-site accuracy than applying the same simulated changes as ordinary data augmentation. Third, the recovered accuracy shows up as better cross-site reproducibility of handcrafted radiomic features computed from the predicted masks.
The motivating evidence is qualitative. The research file's summary of AutoPET Challenge on Fully Automated Lesion Segmentation in Oncologic PET/CT Imaging, Part 2: Domain Generalization, a multi-center PET/CT segmentation challenge, attributes external-center drops partly to skip connections passing site-specific high-frequency content around the bottleneck. Rethinking U-Net: Task-Adaptive Mixture of Skip Connections for Enhanced Medical Image Segmentation shows that the right use of skip information varies by task and dataset, and it learns to route skips adaptively. That paper optimizes in-distribution accuracy and does not ask which skips carry site signal. As far as the file shows, nobody has attributed cross-site failure per skip level or constrained skips specifically for acquisition invariance in the small-data regime. The file's section on closest existing work was truncated, so this gap claim is provisional and must be rechecked against the literature before committing.
Research Program
The first component is a diagnostic we call per-level feature realignment. We train a U-Net on one center. At an external center we have no paired scans of the same patient. Instead, at inference we re-standardize the channel statistics of one skip level at a time so they match the source center's running statistics, leaving all other levels untouched, and measure how much external Dice recovers. Dice is the standard overlap score between predicted and reference masks. We do the same for the bottleneck. On synthetic perturbations, where paired clean images exist, we also swap in the clean-image skip features level by level. The resulting recovery profile across levels is the attribution. If recovery is spread evenly across levels, or concentrated at the bottleneck, the core claim is wrong, and we will report that.
The second component is the fix. We define a family $T$ of acquisition perturbations: nonlinear intensity remapping, bias fields, resolution and slice-thickness changes, blur, and noise at different reconstruction strengths. These mimic the hardware and protocol variation that Automatic Segmentation of Vestibular Schwannoma from T2-Weighted MRI by Deep Spatial Attention with Hardness-Weighted Loss and Computed Tomography Radiomic Nomogram for Preoperative Prediction of Extrational Extension in Papillary Thyroid Carcinoma describe as changing voxel intensity and texture distributions. Training minimizes
$$\mathcal{L} = \mathcal{L}_{seg}(x) + \mathcal{L}_{seg}(t(x)) + \lambda \sum_{l \in S} \frac{\lVert s_l(x) - s_l(t(x)) \rVert^2}{\lVert s_l(x) \rVert^2},$$
where $x$ is a training image, $t$ is a perturbation drawn from $T$, $s_l$ is the feature map sent through skip level $l$, $S$ is the set of skip levels chosen by the diagnostic, and $\lambda$ sets the penalty strength. The first two terms are exactly what augmentation alone would use. The only new ingredient is the third term, applied at selected skips. This makes the comparison against augmentation clean.
The third component is the downstream radiomics audit. We compute radiomic features with PyRadiomics, the extraction pipeline used in Comparison and analysis of multiple machine learning models for discriminating benign and malignant testicular lesions based on magnetic resonance imaging radiomics, which extracted 1,781 features and found that most of the top-ranked ones were wavelet or filter based. Those are exactly the texture-sensitive features we expect to suffer most. For each external case we compute the features from the predicted mask and from the reference mask, and report the per-feature intraclass correlation, a standard agreement statistic. We separate shape, first-order, and wavelet-texture families. We are not arguing that handcrafted features are better than learned ones. A Deep Learning-Based Radiomics Model for Prediction of Survival in Glioblastoma Multiforme, which selected features from both types for a survival model, eliminated every handcrafted feature. Our point is narrower: nomogram-style clinical models still consume handcrafted features, and their external validity depends on mask stability.
The small-sample regime is set deliberately. Following Bria et al., who note that specialized clinical cohorts often contain fewer than 100 cases, we train on 10, 25, and 50 cases from a single center. The small-data shortcut-learning results in Unveiling COVID-19 from Chest X-ray with deep learning: a hurdles race with small data, which reports classifiers latching onto scanner and marker artifacts, suggest the effect should be largest at the smallest sizes. We therefore expect the benefit of the fix to shrink as training size grows. If it does not shrink, the mechanism is something other than small-sample memorization of site texture.
Baselines
The naive reference is a standard 3D U-Net trained on the single-center subset with only geometric augmentation, evaluated unchanged at external centers. It fixes the size of the generalization gap the other methods must close.
The strongest simple competitor is the same U-Net trained with the identical perturbation family $T$ as data augmentation, with no feature penalty. It uses the same training cases, number of epochs, and hyperparameter search budget. It is the most important baseline because it spends the same prior knowledge about scanners. Beating it means a statistically significant gain in mean external Dice, using a paired bootstrap over external cases, across at least two of the three training sizes. Matching it in-distribution is required for the result to count as a win rather than a trade-off.
The strongest architectural competitor is the task-adaptive mixture of skip connections from Rethinking U-Net. It is run with the same augmentation, data, and tuning budget, which tests whether learned skip routing already suppresses site signal on its own. For the vestibular schwannoma task we also include the 2.5D attention network with hardness-weighted loss from Wang et al. as a task-specific reference. For PET/CT we compare against the best reported AutoPET domain-generalization entries. However, the file gives no reported numbers: no reported result found on Lacuna for external-center Dice on AutoPET Part 2, and none for vestibular schwannoma cross-site Dice. We will therefore rerun every competitor ourselves under our protocol rather than quote figures.
The ablations isolate the key ingredient by applying the same penalty at three places: the network output only, the bottleneck only, and all skip levels uniformly. These are compared against the diagnostic-selected skips. A further ablation simply removes the highest-resolution skip, which tests whether the fix is just deleting information.
Why not just…?
Why not just use heavy intensity and resolution augmentation? This is the real threat. Augmentation makes the network's output invariant only on the few training images it sees. With 10 to 50 cases, internal skip features can still encode site texture in ways that happen not to hurt on training data but do hurt at a new scanner. The augmentation-only baseline above is built to settle the question, since it differs from our method by only the penalty term. If it matches us, the method contribution collapses, and what remains is the attribution result and the radiomics audit, which we would report as the finding.
Why not just learn which skips to use, as the mixture-of-skips work does? Adaptive routing is trained to maximize source-center accuracy. At a single center, site texture is a perfectly predictive feature, so nothing pushes the router to drop it. Running Rethinking U-Net under our external evaluation tests this directly. If it closes the gap, our attribution analysis applied to its learned routing weights would explain why.
Why not just harmonize the images or start from a pretrained encoder? Image-level harmonization, such as histogram matching to the source center, cannot undo resolution or reconstruction-kernel differences, and our perturbation family includes both. Pretrained encoders help with small data. Cross-dataset adaptation of voxel-level deep radiomics for predicting survival in inoperable locally advanced NSCLC treated with immunotherapy pretrains on a related cohort and fine-tunes on 37 patients. But a pretrained encoder still feeds its high-resolution maps through the skips. We include a pretrained-encoder U-Net, both with and without our penalty, to see whether pretraining and the penalty add up or overlap.
Proposed Experiments
The primary setting is multi-center PET/CT lesion segmentation in the AutoPET domain-generalization configuration. We train on one center and test on the held-out centers. The secondary setting is vestibular schwannoma segmentation on T2-weighted MRI, trained on one scanner or protocol subgroup and tested on others. Each training size (10, 25, 50) is repeated over five random subsets to expose variance, which dominates at these sizes. The metrics are external Dice, boundary distance (95th-percentile Hausdorff), and the radiomic intraclass correlations by feature family.
We will treat three checks as decisive. The first is the attribution profile: at least half of the recoverable external Dice must come from realigning the top one or two skip levels. The second is the paired comparison against augmentation-only with the same perturbations. The third is whether wavelet-texture radiomic features, which are the ones expected to be most mask-sensitive, show a larger reproducibility gain than shape features. If the attribution is flat, the targeted penalty has no principled basis. If augmentation-only ties, the contribution is diagnostic rather than methodological. Both outcomes would be reported as results.
References
- AutoPET Challenge on Fully Automated Lesion Segmentation in Oncologic PET/CT Imaging, Part 2: Domain Generalization
- Rethinking U-Net: Task-Adaptive Mixture of Skip Connections for Enhanced Medical Image Segmentation
- Automatic Segmentation of Vestibular Schwannoma from T2-Weighted MRI by Deep Spatial Attention with Hardness-Weighted Loss
- Computed Tomography Radiomic Nomogram for Preoperative Prediction of Extrathyroidal Extension in Papillary Thyroid Carcinoma
- Comparison and analysis of multiple machine learning models for discriminating benign and malignant testicular lesions based on magnetic resonance imaging radiomics
- A Deep Learning-Based Radiomics Model for Prediction of Survival in Glioblastoma Multiforme
- Towards AI-driven Next Generation Personalized Healthcare and Well-being
- Unveiling COVID-19 from CHEST X-Ray with Deep Learning: A Hurdles Race with Small Data
- Cross-dataset adaptation of voxel-level deep radiomics for predicting survival in inoperable locally advanced NSCLC treated with immunotherapy