Many medical segmentation networks work in two passes. A first pass draws a rough mask, and a second pass looks again, more carefully, at a thin strip around the rough mask's edge, where most errors sit. Today that strip is a fixed width, set by hand with image erosion and dilation. It is too wide for tiny polyps and too narrow when the first pass is badly off. Polyps are physical bumps on the colon wall, so a single-image depth estimator (a network that guesses how far each pixel is from the camera from one photo) usually shows a sharp change in depth where the polyp's true edge is, even when color and texture are identical to the surrounding tissue. We propose to let that depth signal decide where the second pass looks. The refinement strip stretches from the rough mask's edge toward the nearest strong depth edge. It collapses back to a thin default wherever depth is flat or untrustworthy.
The claim is narrow and testable. Depth is most useful for locating where to refine, and less useful as another input channel mixed in everywhere. A strip shaped by depth should beat both a fixed strip and a strip shaped only by the first pass's own uncertainty, and the gains should appear specifically on lesions with visible surface relief. If the gains instead spread evenly or vanish against an uncertainty-shaped strip, the idea is wrong. The experiments below are designed to show which.
What exists and what is missing
Three lines of prior work bracket the idea. Polyper supplies the method we build on. It splits a coarse polyp mask into a confident interior and a "potential boundary" strip using fixed morphological kernels, then refines the strip with cross-attention from interior features. Its fixed strip width is the limitation we target, and it uses no depth. CaraNet also refines edges from a coarse mask, by inverting the coarse foreground so attention flows to unresolved margins. Like Polyper it works purely from 2D appearance. Polyp-DAM motivates the proposal. It shows that a frozen Depth Anything model helps polyp segmentation: mean Dice on Kvasir-SEG drops from 0.929 to 0.874 without depth. However, it fuses depth uniformly across four resolutions, with no coarse-to-fine localization and no model of when depth is wrong.
Two papers constrain the design. See in Depth (DepSeg) is the closest coarse-to-fine use of depth. It clusters DepthAnythingV2 output to place point prompts for SAM2 in surgical scenes, but it drops depth once SAM2 refines the mask, and it fails on structures with no depth relief such as the hepatic vein. That failure is a warning: any depth-driven refinement must fall back gracefully on flat anatomy, such as flat or sessile polyps. Depth-Guided NeRF Training via Earth Mover's Distance comes from 3D reconstruction. It shows that monocular depth is noisy at object boundaries and on reflective surfaces, and that treating it as a soft, uncertainty-weighted suggestion prevents it from corrupting the output. Its diffusion-based uncertainty is too slow to reuse here, so we need a cheap reliability estimate instead. No More Sliding Window is a third, more distant reference. It shows in 3D CT that a coarse pass can learn where to spend fine computation. We apply the same principle to where the refinement strip goes, not to which patches are processed.
What nobody has done is make the geometry of the refinement region itself a function of depth relief, gated by depth reliability. Polyper-with-depth, Polyp-DAM, and DepSeg all use depth either as a feature or as a prompt source. None uses it to reshape where the fine stage operates.
Core mechanism
A coarse segmenter produces a mask and its boundary. A frozen Depth Anything V2 model produces relative depth. Because relative depth has an unknown scale and offset, we work with the gradient magnitude of per-image normalized log-disparity, $E(p)$ at pixel $p$, which is insensitive to global scale.
Along each boundary point's outward and inward normal, we search up to a maximum distance $d_{max}$ for the strongest peak of $E$. The strip half-width at that boundary point is
$$ w(b) = w_0 + r(b)\, \min\big(d^*(b), d_{max}\big), $$
where $w_0$ is a small default width, $d^*(b)$ is the distance from boundary point $b$ to the depth-edge peak, and $r(b)\in[0,1]$ is a reliability weight. Where there is no depth edge or depth is unreliable, $r\to 0$ and the strip reverts to the thin Polyper-style default.
The reliability weight $r$ combines two cheap signals. The first is the strength of the depth-edge peak relative to local depth texture. The second is the agreement between depth predicted on the image and on its horizontal flip, which costs one extra forward pass. We also mask specular highlights, which are easy to detect from saturated pixels. A learned version replaces the hand-set search with a tiny head that predicts $w(b)$ from local $E$ profiles, supervised by the distance from the coarse edge to the ground-truth edge.
Inside the strip, the fine stage is Polyper's interior-to-boundary cross-attention, with the depth-edge map added as a feature scaled by $r$. Outside the strip, the coarse prediction stands.
As a secondary use of the same machinery, strong closed depth-relief contours that lie entirely outside the coarse mask become extra candidate regions for the fine stage. This lets us measure whether depth can partly recover first-stage misses, the error-cascading problem of coarse-to-fine systems.
Research Program
The work proceeds in three steps. First, we measure how often and by how much the coarse boundary misses the true edge on standard polyp data. We also measure how often a strong depth edge lies between them, which gives a relief score per lesion. This analysis alone says whether the idea has room to work. If depth edges rarely sit near true edges, we stop. Second, we build the hand-set and learned versions of the depth-shaped strip on a shared backbone. Third, we test transfer to a harder domain, laparoscopic surgery, where some classes have strong relief and others none, to see whether the reliability gate prevents harm.
Baselines
The naive reference point is the coarse segmenter alone, with no refinement stage, trained on the same data. It tells us how much any refinement buys.
The fixed-strip baseline is Polyper, reimplemented on our backbone with its morphological strip. We tune its kernel size on validation data so it is not handicapped by a poor default. Its reported ETIS mDice is 86.51%. Because it is the exact architecture we modify, it isolates the effect of changing the strip.
The strongest existing depth-using method is Polyp-DAM, which reports 0.891 mean Dice on ETIS and 0.941 on CVC-ClinicDB. We run it with its released design and the same Depth Anything weights. We also report its published numbers alongside, noting any protocol differences. Beating it means higher mean Dice on the out-of-distribution sets (ETIS, CVC-ColonDB, CVC-300) under an identical Kvasir-SEG plus CVC-ClinicDB training split. We are not willing to claim a win from in-distribution sets alone.
The ablation that isolates our key ingredient is an uncertainty-shaped strip. It uses the same adaptive-width machinery, but $w(b)$ comes from the coarse network's own per-pixel entropy instead of from depth. It gets the same parameter count and tuning budget. A second ablation is Polyper with depth added as an input channel and a fixed strip, which separates "depth as feature" from "depth as geometry." An oracle strip, placed from the ground-truth edge, gives an upper bound on what better strip placement could ever give.
All methods share the backbone, the input resolution, the training schedule, the augmentation, the frozen depth model where depth is used, and an equal hyperparameter-search budget. We report three seeds.
On CholecSeg8k, the reference points are the supervised Mask2Former at 78.4% mIoU and training-free DepSeg at 35.9%, both reported in See in Depth. There we add our strip on top of a supervised segmenter and compare against that segmenter with a fixed strip.
Why not just…?
Why not just feed depth as a fourth input channel, as Polyp-DAM effectively does? This is a strong alternative, and Polyp-DAM's ETIS result shows that fused depth already helps a lot. The risk the file documents is different: raw depth carries blur and halo artifacts exactly at boundaries (Depth-Guided NeRF via EMD). A network fed depth everywhere must learn by itself where to ignore it. The Polyper-plus-depth-channel ablation settles the question directly. If it matches the depth-shaped strip, our claim that geometry-of-refinement matters more than depth-as-feature fails.
Why not just shape the strip from the coarse network's own uncertainty? This is the most serious threat. The coarse network may already "know" where it is unsure, which would make depth redundant. Our bet is that the coarse network is confidently wrong on low-contrast edges, since that is where appearance carries no signal. Depth then carries information that RGB uncertainty cannot. The uncertainty-shaped strip ablation, stratified by lesion relief score, decides it. We expect parity on low-relief lesions and a clear gap on high-relief ones. If there is no gap on high-relief lesions either, we drop the depth strip and report that result.
Why not just fine-tune SAM2 on polyp or surgical images? Zero-shot SAM2 reaches only 14.7% mIoU on CholecSeg8k (See in Depth). Fine-tuning would surely do better, but it still refines edges from appearance alone. That is the regime where low-contrast polyp edges are hard. A fine-tuned SAM2 is a reasonable extra comparison on ETIS. Our strip can also sit on top of any coarse segmenter, including a fine-tuned SAM2, so the two approaches are complementary rather than exclusive.
Proposed Experiments
We train on the standard Kvasir-SEG plus CVC-ClinicDB split and test on ETIS, CVC-ColonDB, and CVC-300. We report mean Dice and IoU, plus boundary-focused metrics: boundary F-score and 95th-percentile Hausdorff distance. Every result is stratified by polyp size and by a per-lesion relief score, computed from depth-edge strength along the ground-truth contour. For recall, we report the fraction of lesions entirely missed by the coarse stage that the depth-relief candidates recover, together with the resulting false positives.
The decisive checks are three. First, gains over the uncertainty-shaped strip must concentrate on high-relief lesions and be near zero, not negative, on flat ones. Second, when we inject synthetic depth corruption (blur, specular-like holes, random affine distortions), the reliability gate must return performance to the fixed-strip baseline rather than below it. Third, on CholecSeg8k, relief-rich classes such as liver and instruments should improve. Depth-neutral classes such as the hepatic vein must not degrade, which directly addresses the failure mode that DepSeg reported.
A final cost check reports added latency from the depth model and the flip pass. This matters because the frozen foundation backbone, not the small refinement head, dominates run time.
References
- Polyper: Boundary Sensitive Polyp Segmentation
- CaraNet: context axial reverse attention network for segmentation of small medical objects
- Polyp-DAM: Polyp Segmentation via Depth Anything Model
- See in Depth: Training-Free Surgical Scene Segmentation with Monocular Depth Priors
- Depth-Guided NeRF Training via Earth Mover's Distance
- No More Sliding Window: Efficient 3D Medical Image Segmentation with Differentiable Top-K Patch Sampling