Segmentation networks shrink images step by step as they go deeper, and each shrinking step averages neighboring pixels together. For a thin structure like the right-ventricular wall in a cardiac MRI, a few pixels across, that averaging blurs the structure into its surroundings. The damage is worst on images from a scanner the model never saw, because there the intensities and pixel sizes the network learned to rely on have changed. Cardiac MRI is acquired as a movie (cine) covering the whole heartbeat. Yet the strongest cross-vendor segmenters discard the movie and segment single frames. The movie carries a signal that does not depend on how a vendor maps tissue to brightness: which pixels brighten and darken together over the cycle, and where neighboring pixels stop moving in lockstep. That second pattern marks a moving boundary.
We propose to compute this signal in a form that is exactly unchanged by any monotonic remapping of intensity, and to use it for one job only: deciding which pixels survive each downsampling step. Concretely, we replace the network's pooling layers with learnable weighted pooling, as in LIP: Local Importance-Based Pooling. The weights come from a map of rank correlations between neighboring pixels' intensity-over-time curves, not from the vendor-dependent image features. The bet is that thin, moving structures are preserved at every scale by a mechanism that, by construction, cannot be fooled by a new scanner's contrast curve.
Core Claim
The claim is narrow and testable. On cross-vendor cine MRI segmentation, gating the downsampling path with an intensity-invariant temporal signal improves thin-structure accuracy on unseen vendors more than feeding the same temporal signal as an extra input channel. It also beats gating with ordinary image-derived importance. The gains should concentrate on the right ventricle, the myocardium, and basal and apical slices.
The mechanism is defined as follows. Let $I_t(x)$ be the intensity at pixel $x$ in frame $t$ of a cine loop with $T$ frames. Let $\rho(x,x')$ be the Spearman rank correlation between the time curves of $x$ and a neighbor $x'$. The motion-boundary map $B(x)$ is one minus the mean of $\rho$ over the 8-neighborhood. Because Spearman correlation depends only on temporal orderings, $B$ is unchanged if every intensity is passed through any strictly increasing function. That function can be global or vary per pixel, which covers gamma, contrast, bias-field-like gain, and vendor-specific windowing. At each downsampling stage, with $F$ the feature map, $W(y)$ the pooling window feeding output location $y$, and $h$ a small convolutional network applied to $B$ resampled to that stage's resolution, the pooled output is
$$O(y) = \frac{\sum_{x \in W(y)} e^{h(B)(x)}\, F(x)}{\sum_{x \in W(y)} e^{h(B)(x)}}.$$
This is LIP's weighted average with one deliberate change. In LIP the weights are computed from $F$ itself. Here they are computed only from $B$, so the decision about what to keep inherits $B$'s invariance even though $F$ does not. Pooling still operates in pixel units. To handle pixel-spacing differences, we compute $B$ after resampling to the nnU-Net target spacing and combine it with standard resolution augmentation, rather than claiming spacing invariance.
Positioning Against Prior Work
Three lines of work each cover two of the three ingredients.
The first line handles temporal information but not scanner shift. Spatiotemporal-Sensitive Network for Microvascular Obstruction Segmentation from Cine Cardiac Magnetic Resonance shows that motion carries information static frames lack. It feeds inter-frame residuals to a second encoder and reaches 0.5556 Dice on a single-scanner, single-site cohort where static UNets do worse. That work motivates ours. We differ in two ways. Its residuals are intensity differences, which scale with vendor contrast. And its motion signal enters as features to be fused, not as a control on downsampling.
The second line handles scanner shift on static images. The M&Ms challenge winner, Studying Robustness of Semantic Segmentation under Domain Shift in cardiac MRI, and Disentangled Representations for Domain-generalized Cardiac Segmentation both work on single frames. The first uses batch normalization plus heavy intensity augmentation. The second uses resolution augmentation plus anatomy/style factor swapping. Both supply the training recipe we build on, and neither touches the cine dimension or the pooling operator.
The third line handles pooling but not medical temporal data. LIP showed learnable importance pooling helps small objects in natural images (COCO small-object AP 21.9 to 24.0). S4ND: Single-Shot Single-Scale Lung Nodule Detection found max-pooling beat average and strided pooling for tiny CT nodules. These two results disagree about which pooling protects small targets. Neither was tested under scanner shift, so we include both as competitors.
Two benchmark papers constrain the design. AMOS and the M&Ms-2 right-ventricle study, Deep Learning Segmentation of the Right Ventricle in Cardiac MRI: The M&Ms Challenge, show that small structures and basal/apical slices fail first under shift. They also show that well-configured plain U-Nets beat elaborate architectures. Any new component must therefore be a drop-in inside nnU-Net that survives its tuning, not a new architecture.
Research Program
We will build the gated pooling layer as a replacement for strided convolutions in the 2D nnU-Net encoder, and train it with the M&Ms-winning recipe of batch normalization and intensity augmentation. The full cine loop is used at train and test time to compute $B$. Only end-diastolic and end-systolic frames carry labels, which is how M&Ms is annotated.
We will then analyze what the gate learns. The analysis has three parts. First, we measure whether $B$ is in practice concentrated on blood-pool/myocardium and right-ventricular boundaries. Second, we measure how much of $B$'s invariance is lost to noise in flat regions. Monotonic invariance does not cover additive noise, so ranks in static regions become random. Third, we check whether this matters, since in flat regions the gate's choice should be inconsequential.
A secondary arm transfers the same layer to echocardiography. We train on EchoNet-Dynamic and test on CAMUS. This is a harder test, because speckle noise attacks rank stability directly.
Baselines
The naive reference is a 2D nnU-Net with default strided convolutions, trained on Vendors A and B with standard nnU-Net augmentation. It shows how much any intervention buys.
The strongest existing method is the M&Ms-winning configuration from Studying Robustness of Semantic Segmentation under Domain Shift in cardiac MRI: 2D nnU-Net with batch normalization and extensive intensity augmentation. On the hard Vendor B→A shift it reports Dice 0.8457, versus 0.6622 for augmentation alone. On unseen Vendor D the most relevant reported right-ventricle number is 0.624 Dice with resolution augmentation, from Disentangled Representations for Domain-generalized Cardiac Segmentation. We will reproduce both recipes ourselves on the same splits rather than compare to quoted numbers, because preprocessing differs across papers. Beating them means two things on Vendor D and on B→A. Right-ventricle and myocardium Dice must improve by more than the across-seed spread, over five seeds. And the Vendor A↔B asymmetry must shrink, with no loss on in-domain Vendors A/B.
The second competitor is a temporal one: a 2D+time nnU-Net that takes a short window of neighboring frames as a third dimension. It tests whether simply giving the network the movie suffices.
The third competitor is a foundation model. We fine-tune MedSAM frame-by-frame with boxes derived from ground truth. This is an optimistic reference, given the cross-scanner robustness reported for SAM in Foundation versus Domain-Specific Models for Left Ventricular Segmentation on Cardiac Ultrasound: 0.857 versus 0.667 Dice on point-of-care ultrasound.
Three ablations isolate the key ingredient. In the first, $B$ is fed only as an extra input channel and pooling is left unchanged. In the second, LIP gates pooling from image features, which is the original LIP. In the third, LIP gates from intensity-difference residuals, in the style of STSN. We also run max-pooling, following S4ND, to address that paper's finding directly.
All variants share the nnU-Net plan, epochs, augmentation, and patch size. Each gets the same budget of tuning configurations, with selection on a held-out Vendor A/B split only, never on C or D.
Why not just…?
Why not just use nnU-Net with batch normalization and heavy augmentation? This is the real incumbent, and the 0.6622 to 0.8457 jump shows how much it fixes. But augmentation only samples intensity transforms the designer thought of. Our gate is invariant to the entire class of monotonic ones. M&Ms-2 and AMOS also show residual failures concentrated on thin and small structures. We do not compete against this recipe; we insert into it. If the gated model trained with the recipe does not beat the recipe alone on right ventricle and myocardium for Vendor D, the idea fails.
Why not just add the motion map as an input channel? This is the most dangerous alternative, because it costs nothing and uses the same invariant signal. Our hypothesis is that an input channel is itself averaged away by the first strided convolutions, just like the thin structure it marks. Gating instead re-applies the signal at every scale. The input-channel ablation decides this directly. If it matches the gated version, we will report that the invariant signal, not the pooling mechanism, is the contribution.
Why not just resample to uniform spacing and use resolution augmentation? It works: 0.474 to 0.624 right-ventricle Dice on Vendor D. But it targets geometry, not contrast, and interpolation blurs thin borders before the network sees them. We treat it as complementary and report all four combinations of with/without resolution augmentation and with/without gating. The combinations show whether the gains overlap.
Why not just run SAM frame by frame? It needs a prompt per frame and has no notion of temporal coherence. Its reported robustness is on echo, not cine MRI. We include an oracle-box version precisely because it might win on Dice. If it does, the relevant comparison shifts to promptless deployment and ejection-fraction consistency across the cycle, which we also measure.
Proposed Experiments and Decisive Checks
The primary benchmark is M&Ms, using the protocol of training on Vendors A and B. Vendor C is held out and unused, so the setting is pure generalization. Vendor D is the unseen test vendor. We also run the A-only→B and B-only→A shifts, where Studying Robustness of Semantic Segmentation under Domain Shift in cardiac MRI documented asymmetry. M&Ms-2 provides a second test of right-ventricle segmentation across nine scanners. There we report short-axis Dice, Hausdorff distance, and right-ventricular ejection-fraction error, which that challenge found more sensitive to basal and apical mistakes than Dice. For context, M&Ms-2 top models reached about 0.94 short-axis right-ventricle Dice. All results are stratified by slice position (basal, mid, apical) and by structure thickness.
The first decisive check is a controlled invariance test. We apply random strictly increasing intensity curves, both global and spatially varying, to Vendor A/B test images. The rank-gated model's pooling weights should be identical by construction, and its Dice drop should be smaller than that of image-gated LIP and the batch-normalization recipe. If they degrade equally, the invariance does not matter downstream.
The second check is temporal robustness. We subsample cine loops to half and one-third of their frame rate. $B$ should change little, because rank correlations survive subsampling of monotone segments. Residual-based motion encoders should change much more, which addresses the frame-rate fragility flagged for residual-frame methods.
The third check is noise. We add Rician noise at increasing levels and plot gate stability against Dice. This tells us whether the method is likely to carry over to noisy acquisitions such as echocardiography, where we run the EchoNet-Dynamic→CAMUS transfer.
The outcome that would kill the proposal is no gain on Vendor D thin structures, or a gain fully matched by the input-channel ablation. The outcome that would confirm it is a gain concentrated on right ventricle, myocardium, and basal/apical slices that persists on top of resolution augmentation and batch normalization.
References
- LIP: Local Importance-Based Pooling
- Spatiotemporal-Sensitive Network for Microvascular Obstruction Segmentation from Cine Cardiac Magnetic Resonance
- Studying Robustness of Semantic Segmentation under Domain Shift in cardiac MRI
- Disentangled Representations for Domain-Generalized Cardiac Segmentation
- S4ND: Single-Shot Single-Scale Lung Nodule Detection
- AMOS: A Large-Scale Abdominal Multi-Organ Benchmark for Versatile Medical Image Segmentation
- Deep Learning Segmentation of the Right Ventricle in Cardiac MRI: The M&Ms Challenge
- Foundation versus Domain-Specific Models for Left Ventricular Segmentation on Cardiac Ultrasound