Distilling Away Boundary Ambiguity: Teacher Relabeling for Fast Low-Contrast Segmentation

A segmentation network labels every pixel at once, and each pixel's label is predicted independently of its neighbors' final labels. In machine translation, this setup is called non-autoregressive generation: all output tokens are predicted in parallel instead of one at a time. Translation researchers learned that such parallel models fail when the training targets admit several equally valid answers. The model averages the answers and produces incoherent output. The standard fix is knowledge distillation, which here means retraining the fast model on a larger model's own outputs. This works because the teacher always picks one consistent answer. We propose that low-contrast medical boundaries, such as a polyp fading into the colon wall, are the same problem in image form. Annotators can legitimately place such a boundary in several places, so a small per-pixel student learns a blurred average. Relabeling the ambiguous boundary zone with one consistent teacher answer should therefore help small real-time students more than current boundary distillation methods do, because those methods pass the teacher's soft, uncertain probabilities straight through.

This matters because the best polyp segmenters today are either heavy iterative models or carefully engineered one-shot decoders. On the heavy side, FANetv2 feeds its predicted mask back for up to three passes and runs at 20.53 FPS with 96.93M parameters. Live colonoscopy needs real-time models; Parallel Res2Net with reverse attention runs at 32 FPS. The proposal turns the translation account of distillation into a measurable, testable account for dense prediction. It yields a concrete recipe and a set of predictions about when the recipe should and should not help.

Core Claim

Our central claim has three parts. First, boundary bands in low-contrast segmentation data carry high label complexity, meaning a high conditional entropy of the label given local image evidence. Second, replacing labels in an adaptively chosen uncertainty band with a teacher's committed hard decisions lowers that complexity. Third, the student's gain should track the complexity reduction, as Zhou et al. showed for translation.

Two predictions follow from that analogy and would be new evidence in vision. The first is a capacity match: a compact student should benefit more from a moderately sized teacher than from the largest one, mirroring Zhou et al.'s finding that small translation students learn better from simpler teachers. The second concerns teacher type. A teacher that resolves the boundary iteratively, committing confident pixels first in the style of MaskGIT, should produce lower-complexity labels than a one-shot teacher of equal size.

The fixed-scale partitioning direction enters as a measurement lens and a design variable, not as a new backbone. Students that pool pixels into coarser fixed patches mix foreground and background inside boundary patches. They should therefore face higher apparent ambiguity, and we predict they benefit more from relabeling.

Relation to Prior Work

Two strands of work motivate the gap. Boundary-aware polyp segmenters attack ambiguity with architecture. MISNet uses parallel reverse and boundary attention branches. CaraNet combines dilated multi-rate feature pyramids with axial reverse attention, which erases confident foreground so later stages focus on the edges. Polyper splits the coarse mask into interior and boundary with fixed-width erosion and dilation, then refines the boundary using interior features. All three treat the ground-truth boundary as a single correct target. They never ask whether the target itself is inconsistent.

Iterative models show that "thinking longer" helps on these boundaries. RecycleNet feeds latent features back over several steps, and FANetv2 feeds back the predicted mask. Neither compresses that behavior into a fast model.

Two papers supply the methods we build on and compare against, both evaluated only on natural images. BPKD separates edge and body distillation using ground-truth edge masks of fixed width, which avoids contamination from neighboring classes at borders. BRD distills pixel-neighbor affinities from multi-scale teacher features. Both transfer the teacher's soft distribution. Our claim is that, at genuinely ambiguous medical boundaries, the soft distribution is part of the problem.

Zhou et al. supply the theory and the complexity metric. Our contribution is to carry their account into two-dimensional, low-contrast segmentation and test whether its quantitative predictions survive.

We should face one constraint head on. Polyper notes that boundary refinement collapses when the coarse mask misses a lesion entirely. Relabeling with a teacher could spread such misses to the student. For that reason we relabel only inside an uncertainty band around the ground-truth contour and keep ground truth everywhere else.

Research Program

Teachers. We build two families of teachers on the same PVT-style multi-scale encoder at three sizes.

The one-shot teacher is a standard decoder with a boundary branch, in the spirit of MISNet.

The iterative teacher uses parallel masked refinement. It starts by committing the confident interior and exterior pixels. It then masks the uncertain band and re-predicts all masked pixels in parallel over $T$ steps (we will try 4 to 8). At each step it commits the most confident fraction under a cosine schedule, as MaskGIT does with image tokens, and conditions each step on the pixels already committed. This lets neighboring boundary pixels coordinate on a single contour instead of each being predicted independently.

Uncertainty band. The band is not a fixed 7-pixel strip as in BPKD. It is the set of pixels near the ground-truth contour where an ensemble of teacher checkpoints disagrees. Its width therefore adapts to how ambiguous each lesion is.

Students. Students are compact one-shot networks run at real-time speed. The first is a Res2Net-based decoder matching the 32 FPS reference. The second is a lightweight patch-token model whose fixed patch size we vary across 4, 8 and 16 pixels.

Complexity measure. To measure label complexity we adapt Zhou et al.'s conditional-entropy metric. We train a small, fixed probe that predicts a pixel's label from the image patch of side $s$ around it. We then define

$$C_s(d) = \frac{1}{|B|}\sum_{p \in B} H\big(q_s(y_p \mid x_{N_s(p)})\big),$$

where $d$ is a labeled dataset, $B$ is the set of pixels in the uncertainty band, $q_s$ is the probe's predicted label distribution, $x_{N_s(p)}$ is the image patch of side $s$ around pixel $p$, $y_p$ is the label at $p$, and $H$ is entropy.

Lower $C_s$ means the labels are more predictable from local evidence. We compute $C_s$ for the original labels and for each teacher-relabeled version, at several patch sides $s$, and correlate the reductions with student gains.

Baselines

Naive reference. The simplest reference point is the compact student trained on ground truth alone with Dice plus cross-entropy loss. Every other student uses the identical architecture, data splits, augmentation, and epoch budget. Every method gets the same number of hyperparameter-tuning trials on a held-out validation fold.

Strongest existing distillation baseline. This is BPKD-style decoupled distillation, matching soft edge and body probabilities from the same teacher using BPKD's ground-truth edge masks. On natural images BPKD reports a 6.95% mIoU improvement on Cityscapes and a 3.87% margin over channel-wise distillation on ADE20K. We found no reported result on Lacuna for BPKD on any polyp or medical benchmark, so we will run it ourselves. We will also run BRD's affinity distillation with the same teachers, as a soft-feature competitor that needs no edge masks.

Accuracy ceilings. We report the strongest heavy models as numbers the student is not expected to beat but should approach. FANetv2 reports DSC 0.9481 and Hausdorff distance 3.19 on CVC-ClinicDB, and DSC 0.9186 on BKAI-IGH. MISNet reports mDice 0.903 on Kvasir-SEG and 0.918 on CVC-ClinicDB. Polyper reports mDice 86.51% on ETIS-LaribPolypDB.

What counts as success. Beating the baselines means the relabeled student exceeds the BPKD-style student on boundary metrics across three seeds and at matched FPS. The boundary metrics are Hausdorff distance and boundary-band Dice. Any interior Dice loss must stay under half a point.

Ablations. Three ablations isolate the key ingredients. The first trains students on one-shot teacher hard labels versus iterative teacher hard labels at equal teacher size, to test whether parallel refinement is what lowers complexity. The second compares the adaptive band with a fixed-width band and with relabeling of the whole image. The third distills soft labels at several temperatures versus hard labels within the same band.

Why not just…?

Why not just use BPKD or soft-logit distillation at the boundary? This is the main threat, and the soft-versus-hard ablation settles it directly. The translation evidence from Zhou et al. predicts that soft targets preserve the teacher's split between candidate answers, which a parallel student cannot represent. If soft edge distillation matches hard relabeling on Hausdorff distance for the small student, our central claim is wrong and we will report it. We further predict the gap shrinks for larger students, because higher-capacity students can absorb more complex targets. Observing that dependence on student size would distinguish our account from a generic claim that "hard labels are cleaner."

Why not just call this pseudo-labeling and move on? The recipe is close to pseudo-labeling, and we do not claim the training trick is new. What is new is the predictive account and the tests of it. If relabeling helps but the gains do not correlate with $C_s$ reduction, and do not show the capacity-matching pattern, then the translation analogy explains nothing. In that case the honest conclusion is plain self-training.

Why not just make the student itself iterative, like RecycleNet or FANetv2? Iterating multiplies latency by the number of steps. FANetv2's 20.53 FPS shows the cost. We will run a two-step version of the compact student as a reference. If it matches the relabeled one-shot student's boundary accuracy at real-time speed, iteration at test time is the better answer, and we will say so.

Why not just use a bigger multi-scale student such as CaraNet? Multi-scale architecture helps small objects; CaraNet reports 74.7% mDice on ETIS. That approach is orthogonal to ours. We will apply relabeling to a CaraNet-sized student as well, to check whether the gain persists or disappears once the student has enough capacity.

Proposed Experiments

Datasets and protocol. We train on Kvasir-SEG and CVC-ClinicDB. We test within those datasets and also on ETIS-LaribPolypDB and BKAI-IGH as unseen datasets, because ETIS is where models drop most (MISNet reports a 16.8% relative drop versus over 38% for competitors). To check that the effect is not specific to polyps, we repeat the study on ACDC cardiac MRI and the TVUS endometrium ultrasound set, both of which have ambiguous boundaries. We found no multi-annotator benchmark in our sources. Ambiguity is therefore measured through $C_s$ and teacher-ensemble disagreement rather than inter-rater spread, and this is a limitation we will state.

Metrics. We report Dice, mIoU, Hausdorff distance, boundary-band Dice, and FPS on one fixed GPU.

Decisive checks. We consider the account confirmed only if all four of the following hold.

First, iterative-teacher labels must reduce $C_s$ more than one-shot-teacher labels of equal size.

Second, across all teacher–student pairs, student Hausdorff improvement must correlate positively with the $C_s$ reduction.

Third, the smallest student must peak with the middle-sized teacher, not the largest.

Fourth, students with coarser fixed patches must gain more from relabeling than students with finer patches. Coarse patches mix foreground and background at boundaries, so they should face more apparent ambiguity, and $C_s$ measured at matching $s$ should predict how much each gains.

Any one of these failing narrows what we can claim. The second failing alone would mean the translation account does not carry over to vision.