A deformable convolution is a convolution whose sampling points can move. Instead of reading a fixed 3×3 square of pixels, a small side-network predicts where each of the nine points should land, so the filter can bend along a vessel or wrap around an organ edge. That side-network makes its prediction from image intensities. In medical imaging, the intensities are exactly what changes when a model moves to a new hospital, scanner or MRI sequence, while the anatomy stays the same. We propose to measure whether the learned sampling positions drift when only the scanner changes, and to train them so they do not. Two constraints do this. When the same anatomy is re-rendered with a different synthetic "scanner style," the sampling positions should stay put. When the anatomy is spatially warped, the positions should move with the warp. The goal is a segmentation backbone that generalizes from a single source site without any adaptation at test time. It should help most on the small, irregular organs where current models fail worst.
This matters because two literatures each hold half the answer. Domain-generalization methods for medical segmentation mostly randomize appearance, and one of the strongest openly concedes that it does not model geometric variation (Adversarial Consistency for Single Domain Generalization in Medical Image Segmentation). Deformable backbones such as DeformUX-Net were built to handle geometric irregularity, but they have only ever been tested within a single distribution. The AMOS benchmark reports a collapse of more than 20% in mean Dice on unseen MRI scanners, with small organs like the adrenal glands and duodenum doing worst (AMOS). This is the setting where a flexible but intensity-driven sampling grid could either help or badly hurt, and nobody has checked which.
Core Claim
The sampling offsets of a deformable layer are an intermediate quantity with a known correct behavior under the two kinds of shift that matter. An appearance change should leave them unchanged. A spatial warp should transport them along with the warp. Enforcing these two properties directly on the offsets should outperform enforcing invariance only on the final prediction, at equal augmentation budget.
The reason is that output-level consistency lets the network satisfy the loss while its offsets differ wildly between views. Later layers can compensate on seen styles, but that compensation breaks on unseen scanners. We expect the benefit to concentrate on thin and irregular structures, where a misplaced sampling point lands outside the organ entirely.
What Exists and How This Differs
Three papers sit closest to this idea, and each leaves the specific question open.
Efficient Deformable Convolutional Prompt for Continual Test-Time Adaptation (EDCP) places a deformable layer on the input image as a prompt. It updates that prompt online through backpropagation and batch-norm statistic matching on the target stream. It is test-time adaptation applied at the input. We instead constrain deformable layers inside the backbone during training and do no test-time optimization.
Domain and Content Adaptive Convolution (DCAC) generates kernel weights from domain and content codes, but keeps the sampling grid rigid. It also needs multiple source domains. Its authors report weaker gains on CT than on MR and fundus images. We keep the weights static, act on where the kernel looks, and work from one source.
DeformUX-Net supplies the method we build on: depthwise 3D deformable convolution with offsets predicted along all three spatial axes, and public code (masilab/deform-uxnet). It motivates the work by leaving cross-site behavior untested. DUNet plays the same role in 2D, showing that offsets track retinal vessels within a single dataset.
Two further papers constrain the design.
Adversarial Robustness via Deformable Convolution with Stochasticity replaces learned offsets with bounded random ones and gains adversarial robustness on natural-image classification. This is a real counterpoint: perhaps offsets should be decoupled from the input rather than anchored. We include random offsets as a baseline. We expect them to hurt thin structures, but the experiment, not the argument, will decide.
TASD regularizes predictions with a dictionary of organ shapes at test time. It states that this breaks down on irregular, topologically variable structures. That limitation is the regime where offset anchoring is meant to help.
Research Program
The work has three parts.
The first is a diagnostic study. We train DeformUX-Net on AMOS CT and render each held-out CT volume under many synthetic styles: random intensity remapping, random convolutions, and style swaps from an adaptive instance normalization (AdaIN) style generator, a method that transfers one image's intensity statistics onto another. We then measure offset drift, the mean displacement of absolute sampling positions between styled and original views, layer by layer. On real target MRI scans we cannot pair views. There we test whether unsupervised statistics of the offset field predict per-scan Dice loss. Two such statistics are the deviation of a scan's offset magnitudes from their source-domain distribution, and the roughness of the offset field. If drift is small and unrelated to failure, the whole premise is wrong, and we report that as the finding.
The second part is the training objective. For an image $x$, a style-perturbed copy $s(x)$, and a smooth random spatial warp $\phi$, let $q_\ell(x, p)$ denote the absolute sampling positions (base location plus grid position plus learned offset) at layer $\ell$ and location $p$. We add two terms to the segmentation loss:
$$\mathcal{L}_{\text{inv}} = \sum_\ell \mathbb{E}_p \, \lVert q_\ell(s(x), p) - \mathrm{sg}[q_\ell(x, p)] \rVert_1, \qquad \mathcal{L}_{\text{eqv}} = \sum_\ell \mathbb{E}_p \, \lVert q_\ell(\phi \circ x, \phi(p)) - \phi(\mathrm{sg}[q_\ell(x, p)]) \rVert_1$$
Here $\mathrm{sg}$ stops gradients through the reference view, and positions are expressed in layer-$\ell$ coordinates. The second term says that sampling a warped volume should visit the warped versions of the original sampling points. This handles the geometric shift that appearance augmentation ignores. We use L1 so that a few large legitimate offset changes, at true boundaries, are not over-penalized.
The third part is an analysis of where the gain goes, broken down by organ size and shape. For the thin structures in MSD Hepatic Vessels, we test whether anchoring preserves the vessel tracking that DUNet and DeformUX-Net report within a single domain.
Proposed Experiments
The main setting is single-source training on AMOS. Models train on CT only and are tested zero-shot on AMOS MRI. A second split trains on the scanners of one AMOS center and tests on scanners never seen in training. We report per-organ Dice and 95th-percentile Hausdorff distance, with the adrenal glands, duodenum, pancreas and esophagus reported as a separate "small irregular" group. For zero-shot AMOS CT→MRI we found no reported result on Lacuna. We will establish every number ourselves under one protocol.
For comparison with the literature, we also run the abdominal CT→MRI protocol of the Adversarial Consistency paper and the multi-site prostate single-source protocol. The file does not identify the dataset behind the Adversarial Consistency CT→MRI numbers, so matching it requires recovering that protocol from the paper. On prostate, the two published setups are not obviously identical: TASD reports 81.1% average Dice and Adversarial Consistency reports 71.42% over five unseen sites. We will follow each protocol separately rather than compare across them.
Baselines
The naive reference is a rigid 3D U-Net trained with standard augmentation, i.e. plain empirical risk minimization. For orientation, AMOS reports 89.04% mean Dice for 3D U-Net on its CT test set, and Adversarial Consistency reports 76.60% for its ERM baseline on CT→MRI. We run it with the same patch size, epochs and augmentation pipeline as every other model.
The second reference is DeformUX-Net trained on CT with no generalization machinery. This isolates whether deformable sampling helps or hurts under shift by itself. Its in-distribution AMOS result is 0.871–0.902 mean Dice.
The strongest existing competitor is the adversarial domain synthesizer of Adversarial Consistency. It hallucinates hard style shifts with AdaIN and preserves anatomy with a patch-contrastive loss, and it reports 85.93% Dice on abdominal CT→MRI against 83.39% for GIN. We will run its synthesizer as the style generator $s$ for every backbone. That makes the comparison "same synthesized views, with or without offset anchoring," not "our augmentation versus theirs." Beating it means a statistically significant mean Dice gain on its own CT→MRI protocol over its reproduced number, plus a larger gain on the small-organ group of AMOS CT→MRI.
On prostate, TASD's 81.1% is the reference. It uses test-time batch-norm updates while we do none, so we report it separately as an adaptation-based upper point. EDCP is likewise test-time adaptation, and the file gives no numbers for it. We treat it as out of regime unless its code allows a same-data run.
Four ablations isolate the key ingredient, all at an identical augmentation and compute budget. First, the same backbone with only output consistency, meaning agreement of predicted masks across views. Second, consistency on intermediate features rather than offsets. Third, style invariance only, without the warp term. Fourth, two degenerate offset choices: all offsets fixed at zero, which recovers a rigid depthwise network, and bounded random offsets in the style of DCS. Every model gets the same hyperparameter search budget, reported with three seeds.
Why not just…?
Why not just train DeformUX-Net with heavy style augmentation and stop there? This is the most serious threat. Augmentation may already make offsets approximately invariant. The diagnostic study answers it directly by measuring offset drift in an augmentation-trained DeformUX-Net. The output-consistency ablation, run with the identical views, answers whether explicit anchoring adds anything. If drift is already negligible and the ablation matches the full method, we will report that the offset losses are unnecessary and that the contribution is the diagnostic.
Why not just use a rigid CNN or large kernels with the same augmentation? DeformUX-Net beats the large-kernel 3D UX-Net in distribution. But AMOS shows a plain 3D U-Net beating Swin-UNetr on CT, so extra flexibility is not automatically better, and under shift it could be worse. The rigid-U-Net and zero-offset baselines, trained with the same synthesizer, settle whether deformability earns its cost once domains change.
Why not just randomize the offsets, as DCS does for adversarial robustness? Random offsets remove the input dependence that could carry scanner artifacts, and that result comes from DCS. Its evidence, however, is from classification on natural images, where one-pixel precision rarely matters. The random-offset ablation, evaluated on adrenal glands and hepatic vessels, will show whether the robustness gain survives when thin structures must be tracked.
Why not just use shape priors at test time, as TASD does? TASD itself notes that its low-rank shape dictionary fails on irregular structures. It also requires test-time optimization. Our method adds no inference-time cost beyond the deformable layers themselves.
Decisive Checks
Three results would confirm the idea. First, offset drift in the unregularized deformable model must measurably predict per-scan Dice loss on real unseen MRI. Second, offset anchoring must beat output consistency at equal budget, most clearly on the small irregular organs of AMOS CT→MRI. Third, the warp term must add a gain beyond style invariance alone.
If the first check fails, the mechanism is not the bottleneck. If the second fails, augmentation already does the job. Either outcome is a clean, publishable negative result about where deformable networks break under scanner shift.
References
- Adversarial Consistency for Single Domain Generalization in Medical Image Segmentation
- DeformUX-Net: Exploring a 3D Foundation Backbone for Medical Image Segmentation with Depthwise Deformable Convolution
- AMOS: A Large-Scale Abdominal Multi-Organ Benchmark for Versatile Medical Image Segmentation
- Efficient Deformable Convolutional Prompt for Continual Test-Time Adaptation in Medical Image Segmentation
- Domain and Content Adaptive Convolution for Domain Generalization in Medical Image Segmentation
- DUNet: A deformable network for retinal vessel segmentation
- Adversarial Robustness via Deformable Convolution with Stochasticity
- Single-Domain Generalization in Medical Image Segmentation via Test-Time Adaptation from Shape Dictionary