Doctors rarely read a single scan. A prostate, brain or heart exam usually combines several MRI "sequences": different contrasts of the same anatomy, acquired minutes apart, often with different slice thicknesses. A 3D segmentation network that stacks these sequences as input channels assumes that voxel $(x,y,z)$ shows the same tissue in every channel. In practice it often does not. The patient moves between acquisitions. Diffusion images warp in-plane. Thick slices in one sequence straddle two thin slices in another.
This note proposes that this cross-sequence misalignment is a concrete, testable cause of a known failure: 3D networks systematically underestimate lesion volume. If a network has learned to call a voxel "lesion" only where the sequences agree, then under misalignment it outputs roughly the overlap of two shifted copies of the lesion. That overlap is smaller than either copy. We propose to measure this "intersection shrinkage" under controlled, physically realistic misalignment. We then fix it with a small fusion module that learns a smooth 3D displacement field, scaled to slice spacing, between each sequence's features and a reference sequence. The module is trained only from the segmentation loss.
Core Claim
The motivating evidence comes from Evaluating deep learning and radiologist performance in volumetric prostate cancer analysis with biparametric MRI and histopathologically mapped slides. That study compared a 3D U-Net on T2-weighted plus diffusion MRI against whole-mount pathology. Median tumor volume was 2.3 mL on pathology, 2.1 mL from radiologists, and 1.2 mL (flexible threshold) down to 0.7 mL (strict threshold) from the network. The paper does not say why. Our claim is that a meaningful part of this gap is geometric rather than semantic: it comes from fusing channels that are not in register.
The claim makes a quantitative prediction. Model a lesion as a sphere of radius $r$ and two sequences offset by distance $d$. Their overlap has volume $\frac{\pi}{12}(4r+d)(2r-d)^2$. As a fraction of the full sphere this is approximately $1 - \frac{3d}{4r}$ for small $d$ (standard sphere-sphere intersection, expanded to first order). An early-fusion network that behaves like an "agreement detector" should therefore show negative signed volume error that grows linearly in $d/r$. Small lesions and thick-slice, through-plane offsets should suffer most.
A network that is robust to misalignment should show a flat curve. So should a network that has instead learned union-like behavior, although union-like behavior would push volume error positive rather than leaving it flat. This gives a falsifiable diagnostic that distinguishes the three behaviors, independent of any proposed method.
Relation to Prior Work
The direction on cross-modal misalignment is almost entirely 2D. Work on visible–thermal detection shows that rigid pixel-wise fusion of shifted inputs causes "feature pollution" and boxes that fit neither modality. Weakly Aligned Cross-Modal Learning for Multispectral Pedestrian Detection and Weakly Aligned Feature Fusion for Multimodal Object Detection address this by predicting per-location offsets. Multi-Modal Pedestrian Detection Via Dual-Regressor and Object-Based Training for One-Stage Object Detection Network regresses a separate box per modality. We borrow the learned-offset idea. We change three things: offsets become a smooth 3D field, their through-plane range is set by slice spacing, and the target is volumetric masks where errors show up as volume bias rather than box shift.
Three medical papers sit closest. Cross-modal attention for multi-modal image registration and its MICCAI precursor Cross-Modal Attention for MRI and Ultrasound Volume Registration operate on full 3D volumes. However, they regress a single rigid 6-degree-of-freedom transform as a separate registration task, using registration targets. We predict a dense non-rigid field inside the segmentation network, with no registration supervision.
Robust Semi-supervised Multimodal Medical Image Segmentation via Cross Modality Collaboration handles misalignment by making latent features consistent across modalities on top of SAM-Med3D adapters. Matching high-level semantics can tolerate shifts, but nothing in it models where a boundary is in each sequence. We expect that to be exactly where margin errors live, and we test this directly against it.
Unified generative adversarial networks for multimodal segmentation from unpaired 3D medical images handles unaligned brain MRI contrasts by translation. It slices the volumes into 2D, which throws away through-plane continuity. Deep convolutional networks for pancreas segmentation in CT imaging shows that this kind of slicing produces fragmented 3D shapes, so we keep the model fully 3D.
CARVE supplies a relevant observation: medical volumes are anisotropic, with redundant neighboring slices and focal findings on one or two slices. That is why our offset budget is anisotropic. CARVE itself assumes one pre-aligned volume and is not a competitor here.
Research Program
The work has two parts. The diagnosis matters as much as the method.
Part one: a controlled misalignment protocol. We take multi-sequence datasets whose sequences are already co-registered and corrupt one sequence with displacements drawn from three families. The first is rigid through-plane shifts of a fraction of the slice spacing, combined with resampling to a thicker slice grid. The second is small rigid rotations. The third is smooth in-plane non-rigid warps resembling diffusion distortion. Each displacement is applied with a known field, so the true per-sequence lesion masks are available in both frames.
We then measure signed volume error against the reference-frame mask for several standard models. We bin results by lesion radius and offset magnitude and check the predicted $d/r$ slope. Run in reverse, the same protocol tests whether a model's output matches the reference mask, the intersection of the two shifted masks, or their union.
Part two: the fusion module. Each sequence gets its own 3D convolutional encoder. We use a plain 3D U-Net, because AMOS found it beat Swin-UNetr (89.04% vs 86.32% mDice on the CT test set), so a heavy transformer backbone is not justified. At each encoder scale, a small head predicts a low-resolution displacement field that maps the auxiliary sequence's features onto the reference sequence's grid. The features are warped by trilinear sampling and then fused.
Two design choices carry the idea:
- Anisotropic offsets. The field is parameterized in millimeters. Its through-plane range is bounded by the auxiliary sequence's slice spacing, so sub-slice offsets are expressible even when one sequence has 3–5 mm slices.
- Training without registration targets. Training uses the segmentation loss plus a smoothness penalty on the field. A forward-backward consistency term asks that warping reference features into the auxiliary frame and back is close to identity. No registration ground truth is used.
We also add an auxiliary per-sequence mask head, in the spirit of the modal-wise regression work. This lets the model represent a lesion's extent in each sequence's own frame before reconciling them.
Baselines
The naive reference point is an early-fusion 3D U-Net that concatenates all sequences as channels, trained on clean co-registered data and tested under the misalignment protocol. This is how most deployed multi-sequence models are built, and it is the model whose shrinkage slope we expect to be steepest. A second naive variant is the same network trained with random misalignment augmentation drawn from the test distribution. Beating the first is easy; beating the second is the real bar for the "just augment" alternative.
The strongest classical competitor is offline registration followed by the same 3D U-Net. We use mutual-information-based affine plus B-spline registration of each auxiliary sequence to the reference. It is tuned per dataset on the validation split, with the same wall-clock budget we give hyperparameter search for our model.
The strongest learned competitor is the Cross Modality Collaboration method from Robust Semi-supervised Multimodal Medical Image Segmentation via Cross Modality Collaboration. We run it in the fully supervised setting on the same splits and corruption draws, with the same epoch budget. We additionally report its semi-supervised 10%-label setting, where the paper reports 76.3% Dice on AMOS CT and 84.3% on AMOS MRI. AMOS is not used in our misalignment protocol, because its CT and MRI scans are not paired in register, so those numbers are context rather than a target.
On BraTS-style data, the file reports best whole-tumor Dice of about 0.80 from The Multimodal Brain Tumor Image Segmentation Benchmark (BRATS). UAGAN reports 81.55% FLAIR Dice with 2.53 mm average surface distance. On MS-CMR, the challenge paper Cardiac segmentation on late gadolinium enhancement MRI reports mean volumetric Dice of about 0.766 for myocardium, 0.891 for left ventricle and 0.822 for right ventricle. There is no reported result on Lacuna for any method under controlled cross-sequence misalignment. Those comparisons must therefore come from our own runs.
Two ablations isolate the key ingredient:
- the full model with the displacement field frozen at zero, which keeps the extra parameters and the per-sequence heads;
- the full model with an isotropic offset range in voxels instead of a millimeter range scaled to slice spacing.
What counts as beating the baselines. The proposal succeeds if it reduces the magnitude of the signed-volume-error slope against $d/r$ by at least half relative to the augmented early-fusion network. It must do so while matching or exceeding Dice and surface distance on clean data. It must also beat offline registration specifically on the thick-slice through-plane corruption family.
Why not just…?
Why not just register the sequences offline with classical tools and then train a normal 3D U-Net? For co-registered brain MRI with modest motion, this may simply work, and we treat it as a real threat. The registration papers above report that intensity-based mutual information struggles across physically different modalities and at large initial errors. Their learned attention reached 3.60 mm surface registration error from a 16 mm start. MRI-to-MRI sequence pairs are easier than MRI-to-ultrasound, though. Our expected advantage is narrower: when one sequence has thick slices, a registration that resamples onto the thin grid cannot recover sub-slice position. A field learned for the task can at least learn where the lesion boundary should go. If offline registration matches us on the through-plane family as well, the methodological claim fails. The diagnostic result would still stand.
Why not just augment training with random misalignments? Augmentation may teach the network to tolerate shifts by blurring or shrinking its predictions, which is exactly intersection-like behavior. Augmentation is therefore not obviously a cure, and the protocol measures this directly. If the augmented early-fusion network shows a flat bias curve and matching Dice, explicit offsets are unnecessary. We would then report that augmentation, rather than a new module, is the fix.
Why not just fuse late, by segmenting each sequence separately and averaging or unioning the masks? Late fusion sidesteps feature pollution, but it gives up what each sequence contributes inside the others' ambiguous regions. Union-style merging should overshoot volume by symmetry with the shrinkage argument. We include probability averaging and union of per-sequence U-Nets as baselines. The $d/r$ curves will show whether they over-, under- or correctly estimate volume.
Why not just rely on latent semantic consistency, as Cross Modality Collaboration does? Making features agree tells the network that a region is "the same lesion." It does not say where that lesion's edge lies in each sequence. We therefore expect it to keep Dice reasonable while still showing a negative volume slope. Running it under the identical protocol settles this.
Proposed Experiments and Decisive Checks
Datasets.
- BraTS multi-contrast MRI is the controlled testbed: its contrasts are co-registered, so injected displacements have exact ground truth.
- MS-CMRSeg provides natural misalignment between separately acquired cardiac sequences. We use it as a real-world test where only the reference-sequence label is trusted.
- FastMRI Prostate provides T2-weighted and diffusion images with slice-level PI-RADS scores. It lacks the voxel masks needed for volume error, so we use it only for slice-level lesion detection under injected misalignment.
Metrics. We report Dice, average surface distance, signed relative volume error binned by lesion radius, and boundary recall in a shell around the true margin. For our model we also report endpoint error between the learned field and the injected field. This checks whether the module actually recovers the displacement rather than learning an arbitrary warp.
First decisive check: does shrinkage exist? It is purely diagnostic. On BraTS, does early fusion show negative volume error that scales roughly linearly with $d/r$, as the overlap formula predicts? If the slope is flat, the central hypothesis is wrong, and the method has no motivation beyond Dice.
Second decisive check: is it the field? The zero-field ablation should recover most of the naive model's slope. The isotropic-range ablation should fail specifically on thick-slice through-plane shifts.
Third decisive check: does the gain transfer? On MS-CMR, whose misalignment is real rather than synthetic, does the model's learned field produce lower boundary error than offline registration? We would report a negative outcome there as plainly as a positive one.
References
- Evaluating deep learning and radiologist performance in volumetric prostate cancer analysis with biparametric MRI and histopathologically mapped slides
- Weakly Aligned Cross-Modal Learning for Multispectral Pedestrian Detection
- Weakly Aligned Feature Fusion for Multimodal Object Detection
- Multi-Modal Pedestrian Detection Via Dual-Regressor and Object-Based Training for One-Stage Object Detection Network
- Cross-modal attention for multi-modal image registration
- Cross-Modal Attention for MRI and Ultrasound Volume Registration
- Robust Semi-supervised Multimodal Medical Image Segmentation via Cross Modality Collaboration
- Unified generative adversarial networks for multimodal segmentation from unpaired 3D medical images
- Deep convolutional networks for pancreas segmentation in CT imaging
- CARVE: Cross-Slice Anisotropic Reallocation of Visual Evidence for Efficient 3D Medical Volume Understanding
- AMOS: A Large-Scale Abdominal Multi-Organ Benchmark for Versatile Medical Image Segmentation
- The Multimodal Brain Tumor Image Segmentation Benchmark (BRATS)
- Cardiac segmentation on late gadolinium enhancement MRI: A benchmark study from multi-sequence cardiac MR segmentation challenge