Segmentation networks are usually scored by pooling every pixel into one number, such as Dice or mean Intersection-over-Union (IoU), the overlap between predicted and true masks. Large regions contain far more pixels than small ones, so a model can score well while missing nearly every tiny object. Design decisions validated on these pooled scores may therefore be wrong for the small structures that often matter most, such as mitotic figures, micro-cysts, or flat polyps. This note tests one such decision. In a U-Net, an encoder-decoder network, skip connections copy high-resolution early features straight to the decoder. Rethinking U-net Skip Connections for Biomedical Image Segmentation found that removing the shallowest skip (called L1) improves Dice and IoU by up to 10% in-domain and 13% under scanner shift. Their explanation is that L1 mostly carries scanner-specific noise. That shallow path is also the only full-resolution route for the spatial detail that small objects need, and the reported gains were measured with exactly the pooled metrics that hide small-object failures.
We hypothesize that the L1 skip carries two separable signals. The first is per-image intensity statistics, such as brightness, contrast, and stain, which change with the scanner and harm generalization. The second is full-resolution spatial structure that small instances depend on. Pruning discards both. Pooled metrics reward the first loss and hide the second. The proposal has two parts. First, re-evaluate skip pruning under size-aware metrics. Second, test a minimal alternative: keep the L1 skip but remove each channel's per-image mean and variance before it reaches the decoder. If the hypothesis holds, this alternative keeps the cross-domain gain and recovers small-object recall. If it fails, we learn that shallow spatial detail is not what small objects rely on, which would also be useful.
Core Claim
The claim is that the benefit of L1 pruning comes mostly from large objects. Under size-stratified, instance-weighted evaluation, pruning should lower recall on the smallest instances even where pooled Dice rises. A moment-normalized L1 skip should match pruning's cross-domain pooled score and beat it on small-instance metrics.
The claim is falsifiable in both directions. If pruning shows no small-object cost under instance-weighted metrics, the first half fails. If photometric augmentation with the full skip closes the gap equally well, the second half fails, and the contribution reduces to the evaluation finding.
Grounding in Prior Work
Two results motivate the study and point in opposite directions.
- Skips are needed for small objects. The fine-grained IoU study Revisiting Evaluation Metrics for Semantic Segmentation evaluated 15 architectures on 12 datasets. It found that skip-based multi-scale decoders such as UNet and UPerNet are needed to avoid small-object collapse under per-image, instance-level, and worst-case IoU. It did not examine individual skip stages.
- The shallowest skip hurts generalization. Wilm et al. showed that L1 is the most domain-sensitive skip, measured by Hellinger distance between feature distributions across scanners. They scored only with pooled Dice and IoU.
Neither paper asks what the other's finding implies. The question is sharpest in histopathology, where Wilm's own mitotic-figure task (MS-MF) is a small-object problem.
The pooled-metric failure is well documented outside this setting, and we reuse these papers' metrics:
- The adversarial-robustness study Evaluating the Adversarial Robustness of Semantic Segmentation showed that class-pooled mIoU overstates robustness because attacks erase small objects on individual images.
- SegmentMeIfYouCan scores each connected anomaly as one test instance (sIoU) rather than pooling its pixels.
- The salient-object work Towards Size-invariant Salient Object Detection (SIEva) computes metrics separately within boxes around each object and averages them.
- Small Objects Matters in Weakly-supervised Semantic Segmentation weights each instance equally in IA-mIoU.
The kidney-tubule study AI models for automated segmentation of engineered polycystic kidney tubules shows the gap concretely. UACANet reaches 0.624 pooled IoU overall and 0.83 IoU on large cysts but fails on the tiniest Zone 1 cysts.
Two lines of work set our controls:
- Losses tilt models toward or away from small regions. The hidden label-marginal biases of segmentation losses proves that cross-entropy has a proportion-matching bias and that log-Dice is biased toward tiny regions. Its RCE-L1 loss reaches 93.5% Dice on CVC-ClinicDB versus 90.5% for DiceCE. Any architectural comparison must therefore hold the loss fixed, and also cross architecture with a size-aware loss.
- Heavier designs exist. GRAFNet uses top-down feedback and orientation-tuned filters for subtle lesions, and gen2seg uses generative decoders that excel on thin structures but fail on small objects at reduced resolution. We do not build on either. Our point is that a far smaller change may settle the question first.
Research Program
Step 1: a size-aware evaluation suite. We will build one suite applied identically to every model. It extracts connected components from the ground truth and assigns them to size bins:
- natural pixel-area bins where no domain bins exist;
- the published Zone bins for the kidney cysts;
- for mitotic figures, which are annotated as small objects, a fixed-radius disk around each annotation.
For every model, the suite reports:
- pooled Dice and IoU, for comparability with Wilm et al.;
- per-image mIoU and its worst 1% and 5% tails;
- instance-weighted IoU in the IA-mIoU style;
- SIEva-style partitioned F-measure;
- instance recall and precision per size bin, where an instance counts as found if its matched overlap exceeds a threshold.
Wang et al. report that worst-case tails often reach 0% because annotators omitted small labels. To keep annotation omissions from driving the conclusions, we will:
- report detection-style recall alongside tail IoU;
- set a minimum component size;
- manually audit a random sample of "missed" small instances in each dataset, and report what fraction were true misses versus missing labels.
Step 2: skip variants. We train U-Nets with the encoder, decoder, and training recipe fixed, varying only the L1 path:
- full skip;
- L1 pruned, as in Wilm et al.;
- L1 moment-normalized: per-image, per-channel instance normalization, followed by a learned per-channel scale;
- L1 replaced by a high-pass-filtered copy, as an alternative way to strip low-frequency domain content;
- L1 replaced by its spatially shuffled self, which keeps the statistics but destroys the spatial structure.
The shuffled control is the key attribution test. If shuffling L1 hurts small objects as much as pruning does, the path's value to small objects is spatial structure rather than intensity statistics. We will also recompute Wilm's Hellinger domain-sensitivity measure on each variant's L1 features. This checks that normalization actually removes the scanner signature rather than assuming it does.
Step 3: loss interaction. Each variant is trained under three losses:
- DiceCE;
- RCE-L1;
- SI-Dice from the size-invariant salient-object work.
This tests whether a size-aware loss can compensate for a pruned skip, or whether the information is simply gone once the path is cut.
Baselines
Full U-Net (naive reference). The naive reference point is a standard U-Net with all skips and DiceCE, trained with the same data, augmentation, and number of steps as every other variant. It establishes how much small-object failure exists before any intervention.
L1-pruned U-Net (strongest competitor). The strongest competitor on this question is the L1-pruned U-Net of Wilm et al.. It is the published method that explicitly trades shallow detail for robustness, and it reports gains of up to 10% in-domain and 13% cross-domain in pooled IoU and Dice. The search found no reported size-stratified or instance-weighted result for L1 pruning on Lacuna, so we will produce that number ourselves. We will run it on Wilm's MS-MF, MS-CCT, MV-MR, and synthetic malaria splits, using their leave-scanner-out protocol, and confirm that our reproduction recovers a pooled cross-domain gain of similar size before drawing any conclusions. Beating it means two things together:
- matching its pooled cross-domain Dice within seed variance;
- exceeding its smallest-bin instance recall and its worst-5% per-image IoU, by a margin larger than the across-seed standard deviation over five seeds.
Photometric-augmentation baseline. The second strong competitor is the full U-Net trained with brightness, contrast, saturation, and stain augmentation. It is the obvious non-architectural fix for scanner shift. Its augmentation strength gets the same tuning budget as the normalization variant's single hyperparameter: a grid of equal size, selected on held-out source-domain data only.
Size-aware losses. RCE-L1 and SI-Dice, each paired with the full skip, are the loss-side competitors.
Ablations. The shuffled-L1 and high-pass-L1 variants isolate the key ingredient: whether removing statistics while keeping structure is what matters.
For a medical reference point, the kidney-tubule benchmark gives UACANet at 0.624 pooled IoU, evaluated with leave-one-tubule-out cross-validation. We will report all variants at Zone 1 under the same cross-validation protocol.
Why not just…?
Why not just use photometric augmentation and keep all skips? This is the most serious threat, because Wilm et al. report that brightness changes alone cause up to an 80% IoU loss, and augmentation directly targets such changes. Augmentation cannot cover every scanner difference, such as optical blur or acquisition-specific texture, but per-image moment normalization also removes only first- and second-order shifts, so the two may behave identically. The augmentation baseline above, tuned with the same budget, settles this. If it matches the normalized skip on both pooled cross-domain Dice and small-bin recall, we will report that outcome. The study's contribution then becomes the size-aware audit of skip pruning, plus the practical recommendation to keep L1 and augment rather than prune.
Why not just prune L1 and add a size-invariant loss to compensate? Losses such as SI-Dice and RCE-L1 reweight gradients toward small regions. Mun et al. report up to 8.9 points of small-object gain from size-balanced training in weakly supervised segmentation. However, reweighting cannot restore spatial information that the architecture no longer receives. Below the stride of the L2 skip, a small object is represented by only a few feature-map cells. The full grid of three skip variants crossed with three losses tests this directly. If pruned+SI-Dice matches normalized+SI-Dice in the smallest bin, our hypothesis about shallow detail is wrong.
Why not just switch to a size-invariant loss and ignore the architecture? Pooled-metric bias is partly a training-objective problem, and RCE-L1's improvement on 3D pancreas tumors (from 43.1% with DiceCE to 47.7%) shows that loss choice matters a great deal. Loss changes, however, do not address domain shift, which is the reason pruning exists. The full-skip+RCE-L1 cell of our grid measures how far a loss alone gets under scanner shift.
Proposed Experiments
Primary experiment. The primary experiment uses Wilm's four multi-domain datasets under leave-scanner-out evaluation. MS-MF is the decisive case because mitotic figures are small.
Secondary experiments. We add the kidney-tubule dataset, where Zone 1 cysts serve as the stress test, and polyp segmentation trained on Kvasir-SEG and tested on CVC-ColonDB, CVC-300, and PolypGen. The polyp benchmarks provide a multi-center shift with naturally varying lesion size.
Generality check. To test whether the effect is specific to medical imaging, we will train on Cityscapes with a U-Net-style decoder over a standard encoder. We will then score small instances with the same size-aware suite, and score road obstacles on RoadObstacle21 with sIoU.
Decisive checks.
- Pruning's cost. Does pruning lower smallest-bin recall relative to the full skip while raising pooled cross-domain Dice? This tests the first half of the claim.
- Normalization's benefit. Does the moment-normalized skip lie above both alternatives on the trade-off between pooled cross-domain Dice and small-instance recall? This tests the second half.
- Attribution. Does shuffling L1 reproduce pruning's small-object loss? This tests whether that loss comes from discarded spatial structure.
- Mechanism. Do the Hellinger distances of normalized L1 features fall to the level of the deeper skips? This tests whether normalization actually removes the scanner signature.
We will report all results over five seeds, with paired per-instance bootstrap confidence intervals for the small-bin comparisons.
References
- Rethinking U-net Skip Connections for Biomedical Image Segmentation
- Revisiting Evaluation Metrics for Semantic Segmentation: Optimization and Evaluation of Fine-grained Intersection over Union
- Evaluating the Adversarial Robustness of Semantic Segmentation: Trying Harder Pays Off
- SegmentMeIfYouCan: A Benchmark for Anomaly Segmentation
- Towards Size-invariant Salient Object Detection: A Generic Evaluation and Optimization Approach
- Small Objects Matters in Weakly-supervised Semantic Segmentation
- AI models for automated segmentation of engineered polycystic kidney tubules
- Do we really need dice? The hidden region-size biases of segmentation losses
- GRAFNet: Multiscale Retinal Processing via Guided Cortical Attention Feedback for Enhancing Medical Image Polyp Segmentation
- gen2seg: Generative Models Enable Generalizable Instance Segmentation