Image registration means finding the spatial warp that lines one scan up with another, for example a patient's brain MRI with an atlas. Probabilistic registration networks also output a per-voxel "how sure am I" map, so a clinician can see where the alignment should not be trusted. The structures where alignment matters most and fails most often are the small ones: thin cortical folds, small nuclei, vessel boundaries. Those are also the structures that a network's downsampling layers blur away before the warp is predicted. Our hypothesis is that this is not only an accuracy problem. If a structure has been erased from the features, the network cannot know it is misaligned there, so its uncertainty map will look confident exactly where it is wrong. That makes the choice of downsampling operator, and of which resolution levels feed the uncertainty output, a calibration decision. Today that decision is made by hand.
We propose to test this directly and then act on it. The test is a controlled study of downsampling operators in a diffeomorphic registration network that predicts per-voxel variance. If the effect is real, we follow it with a small, memory-feasible architecture search restricted to downsampling and decoder connectivity. The search is scored by a proper probabilistic score on displacement rather than by overlap alone. We then use adversarial intensity perturbations as a stress test, and ask a cheaper question than the field usually asks. Instead of paying for adversarial training of a 3D registration model, can a well-calibrated network simply flag the voxels an attacker has corrupted?
Core Claim
In probabilistic registration, how well the predicted variance tracks true alignment error on small structures depends strongly on where and how the network discards resolution. We expect this dependence to be stronger than its effect on overall overlap scores. Two consequences follow if the claim holds. First, a narrow search over resolution-handling choices, scored by negative log-likelihood of displacement, should find networks whose uncertainty is markedly better on small structures at equal parameter count and memory. Second, that same calibration should let the uncertainty map detect adversarially induced misalignment at no training-time adversarial cost.
What is new is the coupling. Uncertainty calibration is the search objective, resolution handling is the searched variable, and adversarial perturbation is the evaluation regime. No prior work in the file treats downsampling as a lever on registration uncertainty.
Grounding in Prior Work
Two results motivate the claim. Hierarchical Uncertainty Estimation for Learning-based Registration in Neuroimaging has the network regress target coordinates with per-axis standard deviations. Its directional aleatoric uncertainty correlates with registration error (Spearman 0.601), whereas Monte Carlo dropout barely does (0.181). This tells us explicit variance outputs are the right object to calibrate, but that paper does not ask how the backbone affects them. NADS searches over distributions of flow-based architectures. It found that max-pooling damages out-of-distribution uncertainty while early average pooling helps. That is evidence that pooling governs uncertainty, though only for image-level likelihoods, not for dense displacement fields. Separately, S4ND and ContextMix document how features smaller than about 8×8 pixels are progressively erased by repeated downsampling.
Three papers supply the components we reuse. Unsupervised learning of probabilistic diffeomorphic registration for images and surfaces predicts a velocity field and integrates it into a fold-free warp by scaling and squaring; it averages 0.2 folding voxels and runs in under one second. We keep that integration so topology is guaranteed regardless of the searched backbone. This also addresses the concern raised in the file that search-favored dilations could cause folding. LIP supplies learnable, content-weighted pooling, which raised small-object AP on COCO from 21.9% to 24.0%. F3RNet supplies the "never downsample" option: a full-resolution stream with factorized 3D convolutions. It reached about 86.8% liver Dice but needed 144³ subvolumes to fit in memory.
Two search papers supply the search machinery and constrain how we use it. SparseMask learns continuous, sparsity-penalized connections from encoder stages to decoder heads in about 18 GPU-hours. Scalable Neural Architecture Search for 3D Medical Image Segmentation shows that sampling two operations per edge with Gumbel-Softmax keeps 3D search inside GPU memory, taking about one day on four V100s. Both optimize deterministic overlap. We change the objective and shrink the search space.
Agentic Neural Architecture Search warns that search often just relocates manual design into the search space, and we accept that critique openly. Our search space is deliberately small and hand-chosen, because the scientific question is narrow. FNAS also uses uncertainty in search, but only as a critic's confidence for skipping candidate evaluations. In our work, uncertainty is the task output being optimized.
On the adversarial side, the file offers two constraining results. Mitigating Adversarial Effects Through Randomization defends classifiers by random resizing and padding at test time. That defense deliberately breaks the pixel correspondences registration depends on. Robustness Tokens avoids full retraining by tuning a few prefix tokens on a frozen transformer. It is tied to token architectures and to keeping those tokens secret. AdLER is the closest bridge: it pairs registration with a full min-max adversarial augmentation network for one-shot segmentation. It therefore pays exactly the training overhead we propose to avoid, and its backbone and downsampling are hand-designed.
Research Program
The base model is a diffeomorphic registration network. Its decoder outputs a stationary velocity field, integrated by scaling and squaring, together with a per-voxel, per-axis log-variance of the resulting displacement. It is trained with an image-similarity term, a smoothness term on velocity, and a heteroscedastic term.
Proper calibration needs displacements whose true values are known. We obtain them by warping real scans with randomly sampled smooth, diffeomorphic deformations, and by adding realistic intensity changes such as bias field, noise, and contrast shifts. The heteroscedastic term on these synthetic pairs is
$$\mathcal{L}_{\text{NLL}} = \sum_{v}\sum_{a\in\{x,y,z\}} \left[ \frac{(u_{v,a}-\hat u_{v,a})^2}{2\sigma_{v,a}^2} + \log \sigma_{v,a} \right],$$
where $v$ indexes voxels, $u$ is the known displacement, $\hat u$ the predicted displacement, and $\sigma$ the predicted standard deviation. Negative log-likelihood is a proper score, so it rewards variances that match actual squared error. This makes it a differentiable stand-in for the rank correlation reported by the hierarchical-uncertainty paper.
The program has three stages, and each stage gates the next.
The first stage is a fixed-architecture study. We hold depth, width, and parameter count fixed and swap the operator used at every downsampling stage: strided convolution, max pooling, average pooling, LIP-style learned pooling, and a factorized full-resolution path in the style of F3RNet. For each variant we measure calibration stratified by structure volume. If calibration on small structures barely moves across these variants, the core claim is false and we stop and report that.
The second stage is the search, run only if the first stage shows a real effect. The search space is limited to two things: which operator each of four encoder stages uses, and the continuous, sparsity-penalized connection weights from each encoder stage to the velocity head and, separately, to the variance head. Treating the variance head's connections as their own choice matters. Our hypothesis predicts that the uncertainty output may need finer-resolution inputs than the mean displacement does. Search uses two-operation Gumbel-Softmax sampling for memory, a penalty on peak activation memory so the full-resolution option is not chosen for free, and a validation objective of NLL plus a Dice term on held-out real pairs.
The third stage is adversarial evaluation of the final models. We do no adversarial training at all. We attack each model with projected-gradient intensity perturbations on the moving image that maximize displacement error. We then measure whether the predicted variance flags the corrupted voxels.
The decisive version of the attack is adaptive. Its objective maximizes displacement error while minimizing predicted variance, so the attacker tries to fool both the warp and the uncertainty map. This is the registration analogue of the expectation-over-transformation concern raised for randomization defenses. If calibration collapses under the adaptive attack, we report detection as non-robust and keep the claim confined to non-adversarial calibration.
Baselines
The naive reference point is Monte Carlo dropout on a standard probabilistic VoxelMorph backbone. It is the cheapest uncertainty a practitioner would reach for. The hierarchical-uncertainty paper reports a Spearman correlation of 0.181 with registration error for this approach, so any serious method should clear it easily.
The strongest existing uncertainty baseline is the hierarchical heteroscedastic coordinate-regression model of Hierarchical Uncertainty Estimation for Learning-based Registration in Neuroimaging. It reports Spearman 0.601 between aleatoric uncertainty and reference error across HCP, ADNI, ABIDE, and OASIS3. Its uncertainty-weighted B-spline fitting raises ABIDE Dice from 0.782 to 0.790. Beating it means three things on the same data and the same reference-error protocol. We need a higher error–uncertainty correlation overall. We need a clearly higher correlation in the smallest-structure stratum, which is where our hypothesis predicts the gap. And Dice must not fall below 0.790.
The strongest accuracy baseline is the Bayesian TransMorph. It reports mean Dice 0.745 with zero folding on the JHU inter-patient brain set. We run it on our splits with its released configuration and the same heteroscedastic head added, so the comparison isolates the backbone. To beat it, we must match its Dice within noise while improving small-structure calibration at no more activation memory.
The resolution-preserving baseline is an F3RNet-style network: a full-resolution stream plus a multi-scale stream, with our variance head. It tests whether "never downsample" already solves the problem. Each run of every method gets the same training data, synthetic-deformation generator, number of epochs, and hyperparameter-tuning trials. For the searched model, we add the search cost on top and report it.
Two ablations isolate what is new. The first is the best uniform operator from stage one, applied everywhere without search. The second is the searched model with the variance head forced to share the velocity head's connections. If the searched model does not beat both ablations, the search adds nothing, and the honest result is a design rule rather than a method.
For adversarial cost, the reference is a registration network adversarially trained with a few-step projected-gradient inner loop, under the same perturbation budget. We found no reported result on Lacuna for adversarially trained 3D deformable registration. We will therefore report its accuracy, robustness, and GPU-hours ourselves as the comparison point for "detect instead of defend."
Why not just…?
Why not just use learned pooling or a full-resolution stream everywhere, by hand? This is the most serious threat to the proposal. LIP's gains and F3RNet's accuracy suggest one sensible manual choice might capture most of the benefit. Stage one exists to settle this. If uniform LIP or the F3RNet-style network matches the searched model on small-structure calibration within its memory budget, we will report that the search is unnecessary and publish the design rule. The search is justified only if the best per-stage combination, or separate connections for the variance head, measurably beats every uniform choice.
Why not just recalibrate the variance after training, for example by fitting one scale factor on a validation set? A single global scale can fix overall over- or under-confidence. It cannot recover information the features no longer contain. If small structures were erased before the variance head, their variance will be wrong in shape and ranking, not just in scale, and rank correlation is unaffected by global rescaling. We apply this post-hoc scaling to every baseline anyway. Our claim is about the stratified rank correlation that survives it.
Why not just use deep ensembles or Monte Carlo dropout? The hierarchical-uncertainty paper's 0.181 versus 0.601 comparison shows that weight-space uncertainty barely tracks spatial alignment error. Ensembles also multiply 3D inference cost by the number of members, whereas the probabilistic VoxelMorph line produces variance maps in a single pass of under one second. We still include a five-member ensemble in the small-structure stratum, because no reported result separates ensembles there.
Why not just adversarially train, or randomize inputs at test time? Random resizing and padding work for classifiers precisely because they destroy spatial alignment, and alignment is the output of registration. Even for classifiers, that defense needed adversarial training to reach its best challenge score: 0.924 versus 0.773 for adversarial training alone. Full projected-gradient training multiplies the cost of every 3D update. We measure that cost directly. The detection approach wins only if it flags most attacked voxels under the adaptive attack, at a small fraction of the adversarial-training GPU-hours. If it fails under the adaptive attack, we will say so.
Proposed Experiments
Brain MRI is the main testbed: the OASIS, ABIDE, and ADHD200 scans used by probabilistic VoxelMorph, the IXI and JHU sets used by TransMorph, and ABIDE with the reference-error protocol of the hierarchical-uncertainty paper. CANDI's 28 labeled regions provide a wide range of structure sizes for stratification.
Our metrics are as follows. For calibration, we report Spearman correlation between predicted standard deviation and error, displacement negative log-likelihood, and the area under the curve for flagging voxels whose error exceeds 1 mm. Each is reported overall and separately for the smallest, middle, and largest third of structures by volume. For accuracy, we report Dice, percentage of folding voxels, peak memory, and runtime.
Four checks are decisive. The first is whether calibration in the smallest-structure stratum varies across downsampling operators in stage one by more than seed-to-seed variance. If not, the hypothesis is rejected. The second is whether the searched model beats both the best uniform operator and the shared-connection ablation in that stratum at matched memory. The third is whether, on ABIDE, the searched model exceeds 0.601 Spearman and keeps Dice at or above 0.790. The fourth is whether, under the adaptive attack that also targets the variance, voxel-level detection remains clearly above chance, at well under the GPU-hours of few-step adversarial training.
We expect search to cost roughly SCNAS's one day on four V100s, given a similarly sized space. A negative answer on the first check is itself useful: it would mean resolution choices can be made for accuracy alone, without fear of blinding the uncertainty map.
References
- Hierarchical Uncertainty Estimation for Learning-based Registration in Neuroimaging
- NADS: Neural Architecture Distribution Search for Uncertainty Awareness
- S4ND: Single-Shot Single-Scale Lung Nodule Detection
- ContextMix: A context-aware data augmentation method for industrial visual inspection systems
- Unsupervised learning of probabilistic diffeomorphic registration for images and surfaces
- LIP: Local Importance-Based Pooling
- F3RNet: full-resolution residual registration network for deformable image registration
- SparseMask: Differentiable Connectivity Learning for Dense Image Prediction
- Scalable Neural Architecture Search for 3D Medical Image Segmentation
- Agentic Neural Architecture Search
- FNAS: Uncertainty-Aware Fast Neural Architecture Search
- Mitigating Adversarial Effects Through Randomization
- Robustness Tokens: Towards Adversarial Robustness of Transformers
- AdLER: Adversarial training with label error rectification for one-shot medical image segmentation
- TransMorph: Transformer for unsupervised medical image registration