Every convolutional segmentation network shrinks its feature maps several times, usually with max pooling or strided convolution. Both operations throw information away. One line of work says this loss is what destroys thin boundaries and small structures, and it replaces pooling with lossless rearrangements that fold each 2×2 block of pixels into extra channels. A second line of work, focused on why medical models fail when moved to a new hospital or scanner, says the opposite: images are full of scanner-specific texture and noise, and a model generalizes only if it learns to ignore them. Taken naively, these views conflict. A lossless downsampler hands every bit of scanner noise to the deeper layers.
We propose a downsampling layer that keeps the spatial details while discarding the scanner's signature. It splits each feature map into an orthogonal set of frequency sub-bands: one smooth band and several detail bands for horizontal, vertical and diagonal changes. No pixel values are lost in this split. Each detail band is then normalized on its own, per image and per channel, and during training its overall gain is randomly perturbed. The layer therefore keeps the spatial arrangement of edges and textures in full and discards only a few numbers per band: the band's mean and spread. Our hypothesis is that those few numbers carry most of the scanner identity, because scanners differ mainly in how much high-frequency energy they produce (sharpening kernels, noise level, acquisition matrix) rather than in where edges fall. In two sentences: lossless downsampling helps small structures but passes scanner style straight through, and style is concentrated in per-band energy statistics. So downsample losslessly into frequency bands and normalize away only those statistics.
Core Claim
The claim is both mechanistic and practical. On the mechanistic side, we claim that cross-vendor shift inside a segmentation encoder appears mostly as a change in per-sub-band feature moments at each downsampling stage, with a smaller change in the normalized spatial patterns. On the practical side, we claim that a downsampler built on this observation does two things. It closes more of the unseen-scanner gap than either pure lossless downsampling or pure anti-aliasing. It also does so with the largest gains on thin and small structures, where lossy downsampling hurts most.
The layer has a short definition. Let $x$ be a feature map with $C$ channels. An orthogonal Haar transform with stride $r=2$ maps it to bands $\{x_b\}$: one low-pass band $x_L$, plus three detail bands in 2D or seven in 3D. The transform is a fixed rotation of pixel unshuffling, so it is exactly invertible. Each detail band is then mapped as
$$ \tilde{x}_b = \gamma_b \, e^{\epsilon_b} \, \frac{x_b - \mu_b(x)}{\sigma_b(x)} + \beta_b, \qquad \epsilon_b \sim \mathcal{N}(0, s^2) \text{ during training, } \epsilon_b = 0 \text{ at test}, $$
where $\mu_b$ and $\sigma_b$ are the per-image, per-channel mean and standard deviation of band $b$, and $\gamma_b, \beta_b$ are learned. The low-pass band keeps ordinary batch normalization. A grouped 1×1 convolution then compresses the stacked bands back to the target width. The layer discards exactly $2C$ scalars per detail band per image. Everything else survives.
Relation to Prior Work
Three works supply the downsampling machinery, and none of them tests domain shift. FusionU-Net uses a "reorganize + group convolution" step, which is pixel unshuffling followed by grouped convolution, inside its skip connections for pathology segmentation. It reports 80.04% Dice on MoNuSeg and 91.05% on GlaS, but only on in-distribution splits. iUNets makes the entire U-Net invertible using learned orthogonal strided convolutions. It reaches 0.850 Dice on BraTS 2018, also in-distribution, and pays a 67–107% training-time overhead. i-RevNet motivates the view that standard networks impose an information bottleneck at downsampling. We reuse their lossless split and grouped compression. We depart from them by deliberately removing a small, chosen set of statistics, so the layer is lossless up to a few scalars rather than fully bijective.
Two anti-aliasing works define the main architectural competitor. ASAP downsamples in the Fourier domain with a windowed spectral crop and gains native robustness on ImageNet-C. Dynamic Gaussian and Gabor filters shows that aliasing appears as phase distortion along object boundaries and reports a 1.8-point mIoU gain on ADE20K. Both remove frequencies above the new sampling limit. In medicine, those frequencies include thin vessels and small organ boundaries. Our layer keeps those frequencies and removes only their global energy. Neither paper was tested on medical images or under scanner shift.
The domain-generalization side supplies both the motivation and the strongest competitors. Causality-inspired single-source DG models an image as anatomy plus acquisition factors and randomizes acquisition appearance with shallow random convolutions (GIN) and spatial blending (IPA). FreeSDG treats low frequencies as "style" and mixes them across views. Both methods act only on the input image and leave standard lossy downsampling in place. Our method acts at every downsampling stage. It also targets the detail bands as a style carrier, which partly contradicts FreeSDG's assumption that style lives in low frequencies. We treat this disagreement as an empirical question, settled by the diagnostic described below.
One result constrains our design directly. Studying robustness under domain shift in cardiac MRI found that switching nnU-Net from instance to batch normalization raised M&Ms B→A Dice from 0.6622 to 0.8457. At first glance this result argues against per-image normalization. Our reading is that instance normalization over whole feature maps discards absolute intensity, which carries tissue identity in cine MRI (for example, the bright blood pool). That is why we keep batch normalization on the low-pass band and apply per-image normalization only to the detail bands. If this reading is wrong, the ablation that applies per-band instance normalization to all bands will show no penalty, and we will report that.
Research Program
The program has three stages. The first is a diagnostic that tests the mechanism before any method is trusted. We take a standard nnU-Net trained on M&Ms vendors A and B, as in the M&Ms challenge, and Haar-decompose its activations at each downsampling stage. On held-out scans from all four vendors, we train two linear probes to predict vendor. The first probe sees only per-band moments: 2C numbers per band. The second sees the moment-normalized band features. If the moment-only probe identifies vendor about as well as the full-feature probe, while the normalized-feature probe falls toward chance, the hypothesis is supported. We also compare which bands carry vendor identity. That comparison is the empirical answer to the FreeSDG low-versus-high-frequency question.
In the same stage we test whether per-band moment discrepancy predicts the transfer asymmetry reported for B→A versus A→B. That paper explicitly left this asymmetry unexplained. We also compute the Phase Error Energy metric from the Gabor-filter paper near cardiac boundaries for each vendor pair. This tells us whether aliasing-induced phase error, as opposed to energy shift, tracks the asymmetry.
The second stage builds the layer in 2D and 3D and drops it into nnU-Net. It replaces every strided downsampling in the encoder and is also applied to skip connections in the FusionU-Net style. Training budget, augmentation and patch size stay at nnU-Net defaults. The perturbation scale $s$ is the only new hyperparameter, and we tune it on source-domain validation data only.
The third stage measures what the layer costs: parameters, FLOPs and wall-clock time against nnU-Net. We make no prediction of the overhead and will report what we measure.
Baselines
The naive reference is default nnU-Net with its standard augmentation, trained under each protocol. Every claimed gain is measured relative to it. The comparison is fair because all variants share the same data splits, epochs, patch sizes and tuning budget.
The strongest multi-vendor competitor on M&Ms is the nnU-Net variant from the cardiac robustness study, with heavy augmentation and batch normalization. It reports 0.8457 Dice on B→A and ranked first at M&Ms. We retrain it ourselves rather than copying its numbers, and we count our layer as beating it if mean Dice improves on the unseen vendor D and on B→A, across three seeds, by more than seed variance. The challenge paper reports that top methods reduced the unseen-vendor drop to about 1.5%, against roughly 6% for baselines. Shrinking that residual gap is therefore the target.
The strongest single-source competitor is GIN+IPA from the causality-inspired paper. It reports 70.37 Dice on cross-center prostate MRI (versus 56.59 for its baseline), 86.31 on abdominal CT→MRI and 85.01 on cardiac bSSFP→LGE. We run GIN+IPA alone, our layer alone, and the two combined, on the same splits. Since our layer acts inside the network and GIN+IPA acts on the input, the combined run is the result that matters most. Beating the competitor means the combination exceeds GIN+IPA alone on all three transfers.
On fundus vessels we also compare with FreeSDG, which reports 0.795 Dice for DRIVE→LES-AV. Thin vessels are the clearest place where a low-pass downsampler should fail and ours should not.
The architectural competitors are ASAP and the DGK/Gabor filter, each swapped into the same nnU-Net downsampling slots. Beating them requires higher unseen-domain Dice specifically on thin structures.
Four ablations isolate the key ingredient. The first is pure pixel unshuffling with grouped compression, as in FusionU-Net, which keeps the details but performs no band normalization. The second is the Haar split with ordinary batch normalization, which tests whether band separation alone helps. The third applies band normalization without the random gain, separating the effect of moment removal from the effect of randomization. The fourth applies per-band instance normalization to all bands, including low-pass, which tests our reconciliation with the BN-over-IN finding.
Why not just…?
Why not just do the band-gain randomization as input augmentation? This is the most serious threat. Randomly sharpening, blurring and re-noising the input with a multiscale Haar gain could imitate most of what our layer does, and GIN already randomizes acquisition appearance. Our argument for acting inside the network is that scanner differences pass through nonlinear early layers and re-emerge at every scale, so an input-only intervention cannot remove them at depth. That argument is only an argument until tested. We therefore include an input-level band-gain augmentation, with matched perturbation scales, as a baseline. The stage-one probes will also show whether vendor identity in deep-stage band moments survives input-level augmentation. If the input augmentation matches our layer, we will report the work as an augmentation result and drop the architectural claim.
Why not just switch normalization, as the cardiac study did? Batch normalization produced the largest single gain on record in the file, a jump of 0.18 Dice on B→A in that study. Our layer keeps batch normalization on the low-pass band, so this is a nested comparison rather than a rival method. The open question is whether band-specific normalization adds anything on top of it, and the BN-plus-augmentation baseline answers that directly.
Why not just anti-alias the downsampling? ASAP shows gains on natural-image corruptions, and anti-aliasing may well help with scanner shift too. However, it removes exactly the frequencies that thin structures need. The Gabor-filter paper itself reports that fixed low-pass filters smooth away contour cues. Our prediction is that ASAP matches us on large organs and loses on the adrenal glands, duodenum and vessels. If ASAP matches us there as well, our claim that detail preservation matters fails.
Why not just use lossless downsampling as-is? If FusionU-Net-style unshuffling already generalizes, normalization is unnecessary. We expect it to worsen unseen-vendor performance relative to nnU-Net, because it passes scanner noise through untouched. The pure-unshuffle ablation tests this, and it is the direct experiment on the tension identified in Gap 2.
Proposed Experiments
We use four benchmarks. The first is M&Ms, in two settings: the official protocol trained on A+B and tested on all vendors including the unseen vendor D, and the single-vendor transfers A→B and B→A. The second is AMOS, trained on CT plus one subset of MRI scanners and tested on the held-out MRI scanners. Here AMOS reports 89.04% mDice for 3D-UNet on CT and a drop of more than 20% on unseen MRI scanners; we report per-organ Dice with emphasis on the adrenal glands, duodenum and pancreas. The third is the single-source prostate, abdominal and cardiac protocols from the causality-inspired paper. The fourth is DRIVE→LES-AV/IOSTAR for vessels. Metrics are Dice and 95th-percentile Hausdorff distance, with three seeds per setting.
Three checks decide the outcome. First, the stage-one probes must show that vendor identity is concentrated in per-band moments; if it is not, the mechanism is wrong regardless of downstream Dice. Second, our layer must beat both the pure-unshuffle and the ASAP ablations on unseen-scanner Dice for small and thin structures in AMOS MRI and LES-AV, which is the regime where the two parent ideas make opposite predictions. Third, the layer must add to GIN+IPA and to input-level band-gain augmentation, not merely substitute for them. Passing all three would show that information-preserving downsampling and domain invariance stop conflicting once the discarded information is chosen correctly. Failing the third would show that the effect is achievable more cheaply at the input.
References
- FusionU-Net: U-Net with Enhanced Skip Connection for Pathology Image Segmentation
- iUNets: Fully invertible U-Nets with Learnable Up- and Downsampling
- i-RevNet: Deep Invertible Networks
- Fix your downsampling ASAP! Be natively more robust via Aliasing and Spectral Artifact free Pooling
- Revisiting Downsampling in Semantic Segmentation: Fighting Aliasing with Dynamic Gaussian and Gabor Frequency Filters
- Causality-Inspired Single-Source Domain Generalization for Medical Image Segmentation
- Frequency-Mixed Single-Source Domain Generalization for Medical Image Segmentation
- Studying Robustness of Semantic Segmentation under Domain Shift in cardiac MRI
- Multi-Centre, Multi-Vendor and Multi-Disease Cardiac Segmentation: The M&Ms Challenge
- AMOS: A Large-Scale Abdominal Multi-Organ Benchmark for Versatile Medical Image Segmentation