Small segmentation networks that could run on an endoscope cart or an ultrasound scanner fail in a predictable place: the soft edge where a polyp fades into the colon wall or a tumor fades into surrounding tissue. The standard remedy is knowledge distillation, which trains the small "student" network to copy the per-pixel probabilities of a large "teacher." At those soft edges, though, the teacher is itself unsure. Existing methods treat that unsureness in one of two contradictory ways. Some throw it away as noise. Others keep it as useful information. Neither asks why the teacher is unsure.
The proposal starts from a simple split. When the teacher is unsure at a boundary, it is either because the image really is ambiguous there (low contrast, blur, a border that experts would draw differently) or because the teacher never learned that kind of case. The first kind of uncertainty is correct and the student should inherit it. The second kind is teacher error and the student should not copy it. Bayesian deep learning already separates the two quantities with an ensemble of teachers. The pixels where members agree the image is ambiguous give aleatoric uncertainty. The pixels where members disagree with each other give epistemic uncertainty. We propose to distill the first and to fall back to smoothed ground truth wherever the second dominates. We then test the sharp prediction this makes: removing epistemic-uncertain boundary pixels from distillation helps, and removing aleatoric-uncertain ones hurts.
Core Claim
In low-contrast segmentation, uniform teacher-to-student matching damages students mainly through the teacher's epistemic uncertainty, not its total uncertainty. Gating distillation on the epistemic part alone should give students that are both more accurate and better calibrated at the boundary than gating on total entropy, uniform ensemble distillation, or boundary masks drawn from the ground truth. The same split explains a contradiction in the literature. Paced-Curriculum Distillation trains a segmentation student at first only on pixels with low teacher confidence-based uncertainty and low boundary label uncertainty, admitting the uncertain pixels later on a schedule. DUCore/DyCON argues, in a semi-supervised setting, that discarding uncertain voxels destroys boundary context, and relaxes their penalty instead. We conjecture that each is right about a different kind of uncertainty. Both use total entropy, so neither can tell the kinds apart.
Mechanism
A teacher ensemble of $M$ members produces per-pixel foreground probabilities $p_m(i)$ for pixel $i$, with mean $\bar p(i)$. Write $H$ for binary entropy. Total uncertainty $H(\bar p(i))$ splits exactly into an aleatoric term $A_i = \frac{1}{M}\sum_m H(p_m(i))$ and an epistemic term $E_i = H(\bar p(i)) - A_i$; the epistemic term is the mutual information between the prediction and the choice of ensemble member. The student with prediction $q(i)$ is trained with
$$ \mathcal{L} = \sum_i w_i \,\mathrm{KL}\big(\tilde p(i)\,\|\,q(i)\big) + (1-w_i)\,\mathrm{CE}\big(\tilde y(i), q(i)\big), \qquad w_i = \exp(-E_i/\tau), $$
The symbols are as follows. $\tilde p(i)$ is the ensemble mean after temperature calibration. $\tilde y(i)$ is the ground truth softened near the contour by spatially varying label smoothing, as in P-CD, because hard labels at fuzzy edges are themselves unreliable. $\tau$ is a single scale set so that a chosen fraction of pixels is down-weighted. Aleatoric softness passes through to the student untouched. Only pixels where the teachers disagree are handed back to the smoothed labels.
Calibration is fitted per distance band from the contour rather than as one global temperature. This addresses the observation that teachers are overconfident at transition zones but not in organ interiors. It is a secondary component and is ablated separately. The teacher ensemble is used only during training, so the deployed student carries no extra cost. For a cheaper single-teacher variant, $E_i$ is estimated from test-time augmentation of one teacher instead of a full ensemble.
Relation to Prior Work
Three works motivate and supply the method. Uncertainty-Aware Retinal Vessel Segmentation via Ensemble Distillation distills a five-model FR-UNet ensemble into one student with a uniform KL loss (EnD-KL) and reports near-ensemble accuracy with better calibration. We reuse its ensemble-teacher setup exactly but replace uniform matching with decomposition-gated matching. The Role of Teacher Calibration in Knowledge Distillation shows, for classification, that teacher calibration error predicts student quality far better than teacher accuracy ($R^2 \approx 0.90$–$0.92$ versus $0.55$–$0.67$). This justifies calibrating $\tilde p$, but that work uses a single global temperature with no spatial structure. Calibrated Uncertainty Distillation argues that ambiguous samples should keep high entropy while confident errors are pruned. That argument operates on text classification and uses one uncertainty signal; our proposal is its spatial, two-signal analogue.
Three works constrain the design. BPKD separates edge and interior distillation using masks cut from crisp ground-truth edges. It shows boundary pixels suffer "contextual leakage," but its masks assume unambiguous annotations and ignore whether the teacher is right at the edge. Boundary and Relation Distillation transfers teacher pixel affinities unconditionally on natural-scene benchmarks. U-Know-DiffPAN uses a teacher uncertainty map to send uncertain regions to hard ground-truth loss and certain regions to feature mimicry. That is the closest structural relative. Its map is total uncertainty on a pan-sharpening task, so under our hypothesis it would also discard the genuinely ambiguous edges that a calibrated student should reproduce. On the boundary side, Polyper and Polyp-PVT are strong full-size networks. They serve here as teachers and as upper references, not competitors, since both add inference-time modules a lightweight student cannot afford.
Research Program
The work has three parts. The first is measurement, done before any method is claimed. On polyp and retinal data we compute $A_i$ and $E_i$ from the teacher ensemble and report how each is distributed across distance-from-contour bands. We also report their rank correlation within the boundary band, and how well each predicts teacher error on held-out images. If $A$ and $E$ are nearly collinear at boundaries, with rank correlation above about 0.9, the decomposition has nothing to act on and the proposal collapses to entropy weighting. We would report that outcome and stop.
The second part is a controlled validation that the two signals mean what we claim. We synthetically lower local contrast in a band around true contours on Kvasir-SEG test images, which should raise $A$ while leaving $E$ roughly flat. We then evaluate the same teachers on out-of-distribution images from ETIS and CVC-ColonDB, which should raise $E$ disproportionately.
The third part is the distillation study itself. Its centerpiece is a two-by-two factorial: at matched pixel budgets we remove the top-$k$% of boundary pixels by $A$, by $E$, by both, or by neither from the distillation term. This directly tests the conjectured resolution of the P-CD versus DyCON disagreement.
Baselines
The naive reference is the same lightweight student trained on ground truth alone with Dice plus cross-entropy, with no teacher. This tells us how much any distillation buys.
The strongest existing competitor is EnD-KL from Uncertainty-Aware Retinal Vessel Segmentation via Ensemble Distillation. It uses the same ensemble we use and already transfers ensemble uncertainty, so any gain we show must come from how the uncertainty is used, not from having it. On FIVES it reports student DSC 0.8817 against the full ensemble's 0.8905. On DRIVE it reports student ECE 0.0178 against the ensemble's 0.0259. We rerun EnD-KL ourselves with identical FR-UNet teachers and student. Beating it means matching or exceeding 0.8817 DSC on FIVES while lowering calibration error measured only inside the boundary band, and improving boundary F-score by more than the standard deviation over three seeds.
The second competitor is BPKD-style edge/body distillation from BPKD, with a globally temperature-scaled teacher. It is the most plausible simple recipe for boundary-focused distillation. No reported result was found on Lacuna for BPKD on any medical benchmark, so we run it on our splits. The third is P-CD from Paced-Curriculum Distillation, adapted to teacher-to-student compression. It reports a 2–4% DSC gain on MICCAI18 but, in the file, no absolute value comparable to our benchmarks. The fourth is fixed teacher-entropy thresholding.
The key ablation replaces $E_i$ with total entropy $H(\bar p(i))$ in the gate, holding everything else fixed. It also includes a random gate that down-weights the same fraction of boundary pixels. These isolate whether the decomposition, rather than merely down-weighting uncertain pixels, drives any gain. The per-band temperature is ablated against a single global temperature.
All methods share teachers, student architecture, data splits, epochs, and a hyperparameter search of twenty trials each on a validation split. For polyps, that split is carved from the Kvasir-SEG/ClinicDB training data.
Why not just…?
Why not just soften the teacher with one global temperature? The Role of Teacher Calibration in Knowledge Distillation makes this a serious threat, since calibration alone moved student accuracy substantially in classification. But one temperature softens confident interiors and confidently wrong edge pixels equally. It cannot turn off imitation where teachers disagree. The global-temperature BPKD baseline tests this directly. If it matches our method on boundary F-score and boundary-band ECE, the decomposition is unnecessary and we will say so.
Why not just train the student with a boundary loss or edge branch, without a teacher? Edge branches like those in MISNet and Polyper's boundary attention add inference-time computation, which defeats the purpose of a small model. A loss-only boundary weighting does not add computation, but it still relies on ground-truth contours that are themselves uncertain in these images. We include a boundary-weighted Dice student as an extra naive point. Boundary and Relation Distillation reports a 3.87% mIoU gain from boundary distillation over its plain lightweight model on Pascal VOC, which suggests the teacher adds something, though on natural images.
Why not just use uniform ensemble distillation, since the ensemble already carries uncertainty? EnD-KL is our main baseline for exactly this reason. Its reported results are on thin vessels, where background dominates. It remains open whether uniform matching copies ensemble disagreement into the student at polyp edges on hard sets like ETIS. The factorial experiment answers this. If removing high-$E$ pixels does not help relative to EnD-KL, the idea fails.
Why not just threshold teacher entropy, as P-CD and semi-supervised methods do? Total-entropy gating is the key ablation. Our hypothesis predicts it underperforms because at low-contrast edges it discards mostly aleatoric pixels, which is DyCON's complaint. If total-entropy and epistemic gating perform equally, the measurement stage will likely have already shown $A$ and $E$ to be collinear, and the simpler method wins.
Proposed Experiments
For polyps, teachers are five Polyp-PVT models trained on the standard Kvasir-SEG plus CVC-ClinicDB split. The student is a MobileNetV2 encoder-decoder, following the lightweight choice in EMKD. Evaluation is on Kvasir, ClinicDB, and unseen CVC-ColonDB, ETIS, and EndoScene. For retinal vessels we follow the EnD-KL protocol on FIVES and DRIVE. BUS ultrasound adds a speckle-dominated regime. Pancreas-NIH is a stretch 3D target for testing whether the test-time-augmentation variant scales where a five-model 3D ensemble is costly.
Metrics are Dice and mIoU, boundary F-score within fixed-width contour bands, 95th-percentile Hausdorff distance, and ECE computed both globally and restricted to the boundary band. For reference, the file reports ETIS mDice of 0.787 for Polyp-PVT and 86.51% for Polyper. These are full-size ceilings the student is not expected to reach; the comparison is among distilled students.
The decisive checks come in order. First, $A$ and $E$ must be separable at boundaries, and the synthetic contrast-reduction and domain-shift tests must move them in the predicted directions. Second, in the factorial, removing high-$E$ boundary pixels must help and removing high-$A$ pixels must hurt, by more than seed variance. We expect the effect to be largest on ETIS and ColonDB, where teacher ignorance should dominate. Third, epistemic gating must beat both EnD-KL and total-entropy gating on boundary F-score and boundary-band ECE at equal tuning budget. Failing the first check ends the project cleanly. Failing the second would mean the P-CD versus DyCON disagreement has some other cause, which is itself worth reporting.
References
- Paced-curriculum distillation with prediction and label uncertainty for image segmentation
- DUCore: Dual Uncertainty-Guided Consistency and Regional Contrastive Learning for Semi-supervised Medical Image Segmentation
- Uncertainty-Aware Retinal Vessel Segmentation via Ensemble Distillation
- The Role of Teacher Calibration in Knowledge Distillation
- TRUST THE UNCERTAIN TEACHER: DISTILLING DARK KNOWLEDGE VIA CALIBRATED UNCERTAINTY
- BPKD: Boundary Privileged Knowledge Distillation For Semantic Segmentation
- Towards Customized Knowledge Distillation for Efficient Dense Image Predictions
- U-Know-DiffPAN: An Uncertainty-aware Knowledge Distillation Diffusion Framework with Details Enhancement for PAN-Sharpening
- Polyper: Boundary Sensitive Polyp Segmentation
- Polyp-PVT: Polyp Segmentation with Pyramid Vision Transformers
- Multi-scale information sharing and selection network with boundary attention for polyp segmentation
- Efficient Medical Image Segmentation Based on Knowledge Distillation