Teaching Strided Downsampling to Keep Faint Boundaries When Only Scribbles Are Labeled

Medical segmentation networks shrink images several times on the way down, then rebuild a full-resolution mask on the way up. A growing line of work replaces fixed shrinking operations such as max-pooling with strided convolutions, meaning learned filters that slide over the image two pixels at a time and so halve its size. The argument is that a learned filter can keep the faint edges that pooling throws away. This proposal starts from a gap in that argument. A learned downsampling filter keeps edges only if the training loss rewards it for doing so. When experts mark structures with quick scribbles inside the object instead of tracing full outlines, which is the usual way to get around the cost of expert annotation, no pixel near the boundary carries a label. The filters that decide what survives downsampling then get no direct signal about boundaries. Our hypothesis is that, in this setting, they drift toward smoothing that helps the object interior and erase the low-contrast edges they were meant to preserve.

We propose to test this diagnosis and then fix it with a label-free training signal aimed at the downsampling path itself. The signal is polyphase consistency. A stride-2 layer can sample its input on four different pixel grids (offset by zero or one pixel in each direction), and the network's mask should not depend on which grid the image happens to fall on. Near a faint boundary, aliasing makes the prediction change when the input shifts by one pixel. Inside the object it does not. Requiring predictions to agree across sub-stride shifts therefore produces a gradient almost entirely at boundaries, which is exactly where scribbles provide none. That gradient flows back into the strided kernels. The method adds no annotation and no inference cost, and it keeps the efficient uniform grid of ordinary strided convolutions.

Core Claim

The claim has two parts. First, the boundary advantage of learned strided downsampling over pooling, shown under full supervision, shrinks or reverses under scribble supervision. The cause is that the downsampling kernels lose their boundary training signal, and the evidence should appear as a measurable shift in their frequency response and a rise in shift-sensitivity at boundaries. Second, a consistency loss over the sub-stride shift lattice restores the advantage. It should beat both fixed anti-aliasing and generic augmentation-consistency, because it targets the specific way strided layers fail on edges that no label covers.

Grounding in Prior Work

Three papers motivate the first half. Striving for Simplicity: The All Convolutional Net showed that pooling can be replaced entirely by strided convolutions without losing recognition accuracy. Non-pooling Network for medical image segmentation (NPNet) carried this into medical segmentation. It reports Dice 0.8397 on CVC-ClinicDB against 0.8229 for U-Net, at about one-fiftieth of the parameters, and attributes the gain to preserved boundary detail. LS-Net and the multi-feature bone-surface ultrasound network make the same argument for thin lines and faint acoustic interfaces. All four train with dense labels, so the boundary pixels themselves teach the filters what to keep. We test whether the benefit survives when those labels are removed.

The closest bridge between downsampling and boundaries is Efficient Segmentation: Learning Downsampling Near Semantic Boundaries. It learns a non-uniform sampling grid that concentrates samples near boundaries and argues that boundary error falls as $O(1/N^2)$ rather than $O(1/N)$. This work constrains our design in two ways. It needs dense masks to train the sampler, and it pays for non-uniform warping at upsampling time. We keep the uniform stride and change only what the strided kernels are trained to preserve.

On the weak-supervision side, several methods motivate the problem and supply baselines. Learning to Segment Medical Images with Scribble-Supervision Alone grows scribbles with random walks, refines them with a conditional random field, and filters them with uncertainty. CycleMix adds mixing augmentation and cycle consistency. ScribFormer argues that CNNs under scribbles activate only object interiors and adds a transformer branch as a global shape prior. All three operate on the label or prediction side. None asks whether the encoder's downsampling discards boundary evidence before any decoder or prior can use it.

Boundary Perception Guidance supervises boundaries under scribbles using an off-the-shelf edge detector and comes within 0.4 mIoU of full supervision on PASCAL VOC. The research file notes that such detectors fire on speckle and texture in low-contrast scans. This is the main reason we want a boundary signal that comes from the network's own consistency rather than from image gradients. Boundary-focused fully supervised methods such as MISNet and the reverse-attention polyp network serve as upper references, since they rely on dense masks to extract boundary targets.

Research Program

The first stage is diagnostic. We train one encoder–decoder in three downsampling variants: max-pooling, a fixed low-pass blur followed by stride, and NPNet-style learned stride-2 convolutions. Each variant is trained under full masks and under scribbles, with identical backbone width, schedule, and augmentation. For every variant we measure two things. One is boundary accuracy, via 95th-percentile Hausdorff distance and boundary F-score within a 2-pixel tolerance. The other is a phase-sensitivity score: the average disagreement between the mask on an image and the masks on its sub-stride shifted copies, computed separately in a band around the true boundary and in the interior. We also take the 2D Fourier transform of every learned downsampling kernel and track the fraction of energy above half the Nyquist frequency over training. The hypothesis predicts that under scribbles the kernels lose high-frequency energy relative to the fully supervised run, boundary phase-sensitivity rises, and the boundary metrics of learned stride converge toward those of max-pooling.

The second stage adds the loss. Let $f$ be the network, $x$ an image, $T_\delta$ a translation by offset $\delta$, and $S$ the total stride of the encoder. The polyphase consistency loss is

$$\mathcal{L}_{\text{phase}} = \mathbb{E}_{\delta \in \{0,\dots,S-1\}^2} \, \mathrm{KL}\big(\,\mathrm{sg}[f(x)] \,\big\|\, T_{-\delta} f(T_\delta x)\big),$$

where KL is the per-pixel Kullback–Leibler divergence averaged over the image, $\mathrm{sg}$ stops the gradient through the unshifted prediction, and padding effects are masked out. Shifts that are multiples of $S$ leave every downsampling grid unchanged, so restricting $\delta$ to the sub-stride lattice concentrates the loss on the aliasing behavior of the strided layers. The total objective is partial cross-entropy on scribbled pixels plus $\lambda \mathcal{L}_{\text{phase}}$, with one $\delta$ sampled per batch to keep the cost at one extra forward pass.

We also test a stage-local variant. At each stride-2 layer, the decoder is fed each of the four polyphase subsamplings of that layer's dense output, which is the same as moving the stride grid, and the resulting masks are required to agree. This variant gives each kernel its own gradient rather than one shared through the whole network.

Baselines

The naive reference point is a plain U-Net with max-pooling trained with partial cross-entropy on scribbles only. Every other method has to clear it on boundary metrics, not just Dice.

The strongest existing method on the shared cardiac benchmark is ScribFormer, which reports Dice 0.888 on ACDC with scribbles. Learning to Segment Medical Images with Scribble-Supervision Alone reports 88.0% on ACDC and 78.1% on NCI-ISBI prostate. CycleMix reports 84.8% on ACDC and 80.0% on MSCMRseg. These papers use different splits and post-processing, so we rerun ScribFormer and CycleMix from released settings on our splits with the same epoch budget and a matched hyperparameter search of about 20 trials each. Quoted numbers serve only to check that our reruns are faithful. Because ScribFormer adds a transformer shape prior that is orthogonal to our change, we also add our loss on top of ScribFormer's CNN branch. Beating it means a gain in 95th-percentile Hausdorff distance and boundary F-score at matched or better Dice, significant across five seeds.

Two ablations isolate the key ingredient. The fixed blur-then-stride variant tests whether aliasing control alone, with no learning, is enough. A generic-consistency variant uses random translations of arbitrary size plus flips, with the same KL loss and the same compute. It tests whether restricting shifts to the stride lattice matters, or whether any augmentation consistency does the same job. As an edge-prior competitor, we run the BPG boundary regression branch from Boundary Perception Guidance with a classical edge detector on the same backbone. Fully supervised NPNet and MISNet are upper references. NPNet reports Dice 0.8397 on CVC-ClinicDB, and MISNet reports mean Dice 0.918 on CVC-ClinicDB and 0.903 on Kvasir-SEG. For Embarrassingly Simple Scribble Supervision for 3D Medical Segmentation, no reported result was found on Lacuna for the 2D benchmarks used here. We include its recipe as a 3D sanity check on ACDC if time allows.

Why not just…?

Why not just put a fixed anti-aliasing blur before every stride? This is the most serious threat. If faint boundaries are lost mainly to aliasing, a fixed low-pass filter may fix most of the problem with no learning at all. Against it, the research file notes that strict low-pass filtering also smooths away low-contrast edges, which are the very signal at stake. The blur-then-stride arm of the diagnostic answers the question directly. If it matches our method on boundary F-score in ultrasound and polyp data, we will report that learned downsampling is unnecessary under scribbles, and the paper becomes a negative result about learned strides.

Why not just use standard augmentation consistency, as CycleMix effectively does? CycleMix already enforces consistency under mixing and occlusion. Random translations visit all stride phases in expectation, so this is also a real threat. Our argument is that most of a random shift's consistency signal is spent on grid-aligned components that cannot expose aliasing. The matched-compute generic-consistency ablation settles it. If restricting shifts to the stride lattice gives no boundary gain over random shifts, we drop the specificity claim.

Why not just downsample less, or learn where to sample? Keeping high resolution or using boundary-adaptive sampling would sidestep aliasing. The cost is more compute in the first case, and in the second case a sampler that needs dense masks to train, which scribbles do not provide. We report a variant with one fewer downsampling stage and its cost in operations, so readers can see the accuracy–compute trade.

Why not just add an edge-detector boundary loss? Boundary Perception Guidance works on natural images, where edges are luminance steps. In ultrasound and colonoscopy, classical detectors respond to speckle and folds. The BPG arm on transvaginal-style ultrasound and polyp data will show whether this failure appears in practice.

Proposed Experiments

Benchmarks span increasing boundary difficulty. ACDC cardiac MRI and MSCMRseg use public scribbles from the CycleMix line of work. NCI-ISBI prostate MRI has faint capsule boundaries. Kvasir-SEG and CVC-ClinicDB contain sessile polyps that blend into the mucosa. ISIC2018 skin lesions are included as a moderate-contrast control. For polyp and skin data, scribbles are synthesized from the masks by eroding them and keeping a random skeleton segment, and we check robustness to scribble length.

Metrics are Dice, 95th-percentile Hausdorff distance, boundary F-score at 1- and 2-pixel tolerance, the boundary-band phase-sensitivity score, and the high-frequency energy of the learned kernels.

Three checks are decisive. If learned strides keep their full-supervision boundary advantage under scribbles, the diagnosis is wrong and the loss has no reason to exist. If the polyphase loss lowers boundary phase-sensitivity but not Hausdorff distance, aliasing is not the main cause of boundary error, and confirmation bias in labels or missing shape priors matters more. If the gain grows from ISIC to prostate to polyps, the method is doing what it claims, namely recovering boundaries in proportion to how faint they are.