Segmentation networks for CT and MRI lose much of their accuracy when a scan comes from a different scanner, sequence, or dose level. The damage concentrates at organ and tumor boundaries, where a few voxels of disagreement decide both the Dice score and the clinical usefulness. A separate line of work, robust neural architecture search, adds a robustness term to the objective used to choose a network's wiring. It is based on the observation that some architectures are intrinsically less sensitive to input perturbations than others. Its best-known tool is a penalty on the network's input sensitivity, the Jacobian (the matrix of how much each output changes when each input pixel changes). That penalty is blunt for segmentation. It rewards networks that are insensitive to every change in the input, including the real intensity edges that define a boundary.
We propose a narrower quantity. It is the sensitivity of the network's predictions in a thin band around the predicted boundary, measured only along "nuisance" directions: changes in contrast, noise, and bias field that a different scanner would introduce without moving any anatomy. We use this quantity as an explicit term in architecture search over the volumetric operations that medical search methods already use (2D, 3D, pseudo-3D, and dilated convolutions). We then test whether the architectures it selects hold up better under real, unseen scanner and sequence shifts than a strong conventional pipeline trained with heavy augmentation. We also test whether the gain comes from the architecture itself rather than from the penalty acting on the weights.
Core Claim
Robustness for segmentation should be measured in the right place and along the right directions. A search objective that penalizes boundary-band sensitivity to scanner-like nuisances will select different volumetric architectures than a clean-Dice search or a global Jacobian penalty. Those architectures will shrink the drop in Dice from in-domain to shifted data without the boundary blurring that an undirected smoothness penalty would cause. Concretely, for architecture parameters $\alpha$ and network output $f_\alpha(x)$ on scan $x$, we define
$$S_B(\alpha) = \mathbb{E}_{x}\,\mathbb{E}_{v \sim \mathcal{N}(x)} \left\| M_B(x) \odot J_\alpha(x)\, v \right\|^2 ,$$
Here $J_\alpha(x)$ is the Jacobian of the voxel-wise logits with respect to the input, $v$ is a unit-norm nuisance direction drawn from a generator $\mathcal{N}(x)$ of label-preserving appearance changes, and $M_B(x)$ is a mask selecting voxels within a few millimetres of the current predicted boundary. The Jacobian-vector product $J_\alpha(x) v$ costs about one extra forward-mode pass, so the proxy stays cheap even on 3D patches.
This differs from existing robust search in two specific ways. DSRNA penalizes an undirected Jacobian norm, and RACL bounds a global Lipschitz constant. Both were built for classification margins, and both penalize sensitivity to anatomy as much as to noise. FlatNAS and CRoZe measure flatness and consistency under generic weight or pixel perturbations on CIFAR-scale classifiers. None of them localizes the measurement to where segmentation actually fails, and none restricts it to directions that should not change the answer.
Research Program
The search space is the per-layer operation choice of V-NAS, which picks 2D, 3D, or pseudo-3D convolutions inside a U-shaped encoder-decoder. We extend it with the dilated 3D option used in the fine stage of C2FNAS. We choose this space deliberately. It is small, it has been searched on 3D CT within about 1.2 V100 GPU days per fold, and it isolates the design choice most tied to anisotropic, thick-slice data: whether a layer mixes information across slices. Our hypothesis is testable in this space. Through-plane mixing on thick slices amplifies slice-dependent intensity noise into boundary jitter, so the robust search should prefer pseudo-3D or 2D operations in early high-resolution layers and 3D operations deeper in the network, where features are coarser.
The search is bi-level, as in V-NAS and DSRNA. The inner loop trains weights on the ordinary Dice plus cross-entropy loss only. The outer loop updates $\alpha$ to minimize validation Dice loss plus $\lambda S_B(\alpha)$. Keeping the penalty out of the inner loop is the key design decision: it makes any robustness gain attributable to the chosen wiring rather than to regularized weights. After discretization, the final network is retrained from scratch with the standard pipeline and no robustness term.
The nuisance generator has three families. The first is global style changes produced by perturbing singular values, as in StyCona; we keep perturbations below the strength that paper found breaks label consistency. The second is physics-motivated noise: signal-dependent noise for low-dose CT and Rician noise for MRI. The third is smooth multiplicative bias fields. Search uses only a subset of these, and evaluation uses real shifts the generator never produced. This separation guards against a network that has simply memorized its test corruption.
The boundary mask is recomputed from the supernet's own predictions at each outer step. Using the ground-truth boundary would leak labels into the robustness term. Because the Dice term in the same objective rewards sharp boundaries, a network that lowers $S_B$ by blurring everything pays for it in Dice. That coupling is how the objective addresses the tension between boundary precision and smoothness raised by boundary-focused designs such as HF-UNet and the wavelet edge loss of Yu et al..
Three prior results motivate the program. Kamann found that architectures chosen for clean accuracy are fragile under blur and noise. Mok et al. linked intrinsically smoother architectures to robustness. FlatNAS reports that retraining baseline architectures with sharpness-aware training did not close the gap to its searched ones, cutting CIFAR-10-C mean corruption error from 34.58% to 26.58%. The Medical Segmentation Decathlon analysis (Antonelli et al.) is the main constraint. It found that a well-configured plain U-Net pipeline (nnU-Net) wins broadly, so architecture effects in this domain may be small. We treat that as the hypothesis to beat, not one to explain away.
Baselines
The naive reference is nnU-Net as described in the Decathlon paper. It uses a standard U-Net whose depth, patch size, and resampling are set automatically from dataset properties, trained with its default augmentation on the source domain and applied unchanged to the shifted domain. It matters because it is what a practitioner would deploy. The Decathlon reports large-organ Dice of 0.94 but pancreas tumor Dice of only 0.21, which shows the headroom on hard targets. No reported result was found on Lacuna for nnU-Net under the specific cross-sequence or cross-dataset shifts we use, so we run it ourselves.
The strongest existing competitor for the shift question is nnU-Net trained with StyCona augmentation. StyCona reports 73.39% Dice for bSSFP-to-LGE and 81.59% for LGE-to-bSSFP on cardiac MRI (MS-CMR). The comparison is fair only if StyCona's own reported numbers are reproduced first. We then give the searched architecture the same augmentation, so we test whether architecture adds to augmentation rather than replaces it. Beating this baseline means a statistically reliable gain in shifted-domain Dice, together with no increase in HD95 (the 95th-percentile surface distance, a boundary-error measure), across five seeds.
The strongest accuracy-driven search baseline is a clean-objective search in the identical space, which amounts to V-NAS with the dilated option added. V-NAS reports 85.15% Dice on NIH Pancreas in-domain. This baseline shows whether our objective changes which architecture is chosen at all. A second search baseline replaces $S_B$ with DSRNA's undirected Jacobian norm over all voxels. This ablation isolates our two ingredients, the boundary mask and the nuisance directions. We also run a mask-only and a direction-only variant.
The ablation that most directly threatens the claim puts $S_B$ into the weight training of the fixed nnU-Net architecture and does no search. It shows whether the robustness comes from wiring or from the penalty. All methods get the same training data, the same final-training epochs, and the same hyperparameter budget. We allow twenty trials for $\lambda$ and for each regularization weight, selected on a source-domain validation split only, so no target data is used for tuning.
Why not just…?
Why not just train nnU-Net with heavier augmentation? This is the most likely way the proposal fails. nnU-Net's augmentation and StyCona are strong, and the file offers no medical-imaging evidence that architecture choices survive them. Only the FlatNAS result on CIFAR-10-C suggests that searched structure adds robustness beyond weight-level training. We therefore make augmentation part of every arm. The claim is accepted only if the searched architecture improves shifted-domain Dice on top of StyCona augmentation. If the gap is within seed variance, we will report that architecture adds little once augmentation is strong, which would itself be a useful answer to the question the Decathlon raised.
Why not just add the boundary-band penalty to the weights of a fixed U-Net? This could plausibly work as well, which is exactly why it is a baseline rather than a footnote. The penalty is a differentiable loss and can regularize any network. The weight-regularized nnU-Net arm settles the question. If it matches the searched architecture, the contribution becomes the penalty itself, a segmentation-specific robustness loss, and not the search. We would reframe the result accordingly.
Why not just use a global Jacobian or Lipschitz penalty, as robust search already does? An undirected penalty makes the network less sensitive to real anatomical edges too. The boundary-precision gains reported for HF-UNet (surface distance from 3.892 mm to 1.366 mm) and for the wavelet edge loss show how much those edges matter. The DSRNA-style ablation measures this directly. We expect it to lower sensitivity but raise HD95 and surface distance in-domain. If it does not, the directional restriction is unnecessary and we will say so.
Why not just refine uncertain boundary regions at test time, as DGRNet does? DGRNet finds uncertain regions from disagreement among lightweight adapter heads and refines them using radiology report text. It reports +2.4% Dice and 11% lower HD95 on TextBraTS. It depends on paired reports and on keeping the adapter heads from collapsing into identical predictions, and it works downstream of whatever backbone it is given. It is complementary, not competing, so we do not run it as a primary baseline. We will, however, check whether our boundary-band sensitivity map correlates with disagreement-style uncertainty. Such a correlation would support the idea that the proxy tracks where errors happen.
Proposed Experiments
The first experiment is a gating check that runs before any search. We sample about forty architectures from the space, train each normally on MSD Pancreas, and compute $S_B$ at the end of training. We then measure each architecture's Dice drop when tested on NIH Pancreas, using the organ label only. We compare the Spearman rank correlation between $S_B$ and the Dice drop against the correlations for the undirected Jacobian norm and for clean validation Dice. If $S_B$ does not predict the drop better than these, we stop and report a negative result. CRoZe's reported correlation of about 0.97 with trained-model robustness on CIFAR sets an optimistic ceiling, not an expectation.
If the gate passes, we run the full search on three shift settings. The first is MSD Pancreas to NIH Pancreas, a cross-dataset CT shift. The second is MSD Prostate to Promise12, a cross-dataset MRI shift; both datasets appear in the file via the Decathlon and MS-NAS. The third is MS-CMR bSSFP to LGE and back, a cross-sequence shift with thick, anisotropic slices. On MSD Lung and Liver tasks we add held-out synthetic shifts: low-dose noise, thick-slice resampling, and nuisance families excluded from search. Metrics are Dice, HD95, and average surface distance, measured in-domain and shifted. The robustness gap, in-domain Dice minus shifted Dice, is the primary endpoint, and we also report Dice restricted to the boundary band.
The decisive outcomes are three. The searched architecture must reduce the robustness gap relative to both nnU-Net+StyCona and the clean search, without worsening in-domain HD95. It must beat the weight-regularized nnU-Net, which establishes that the architecture matters. And the choice of operation per layer must differ systematically from the clean search in the direction predicted by the anisotropy hypothesis. We also report search cost against V-NAS's 1.2 GPU days per fold. The added forward-mode pass should keep the overhead in the range A2M reports for its architecture-space method (Pittorino et al., 1.1–1.3x).
References
- DSRNA: Differentiable Search of Robust Neural Architectures
- Adversarially Robust Neural Architectures
- FlatNAS: optimizing Flatness in Neural Architecture Search for Out-of-Distribution Robustness
- Generalizable Lightweight Proxy for Robust NAS against Diverse Perturbations
- V-NAS: Neural Architecture Search for Volumetric Medical Image Segmentation
- C2FNAS: Coarse-to-Fine Neural Architecture Search for 3D Medical Image Segmentation
- Style Content Decomposition-based Data Augmentation for Domain Generalizable Medical Image Segmentation
- HF-UNet: Learning Hierarchically Inter-Task Relevance in Multi-Task U-Net for Accurate Prostate Segmentation in CT Images
- A multi‐class COVID‐19 segmentation network with pyramid attention and edge loss in CT images
- Benchmarking the Robustness of Semantic Segmentation Models
- AdvRush: Searching for Adversarially Robust Neural Architectures
- The Medical Segmentation Decathlon
- DGRNet: Disagreement-Guided Refinement for Uncertainty-Aware Brain Tumor Segmentation
- MS-NAS: Multi-scale Neural Architecture Search for Medical Image Segmentation
- Architecture-aware minimization (A2M): how to find flat minima in neural architecture search