In colonoscopy video, polyps often have the same color and texture as the surrounding bowel wall, so their edges are hard to see in any single frame. Two families of models each solve half of this problem. Single-frame models that also see depth find these edges much better, because a polyp is a physical bump even when it is visually camouflaged. Streaming video models keep depth stable over time by penalizing frame-to-frame change. These two fixes pull against each other. A temporal penalty applied everywhere smooths away the sharp geometric change at the lesion margin, and that margin is exactly the cue the segmentation needs. We propose one streaming network that predicts both the lesion mask and the depth. Its central rule is that the predicted lesion boundary decides where depth is forced to stay smooth over time and where it is allowed to change. Depth is held steady across frames on the polyp interior and on the bowel wall. Inside a narrow band around the predicted lesion margin, the network is instead asked to keep a crease in the surface, meaning a change in surface orientation, and to carry that crease forward from frame to frame.
The opportunity is concrete. EndoStreamDepth reports that its temporally stable streaming depth reaches $\delta_1 = 0.952$ on the phantom-colon benchmark C3VD, while noting that enforcing edge sharpness worsens flicker. Polyp-DAM shows that removing frozen depth guidance drops Kvasir-SEG Dice from 0.929 to 0.874. No system in the file lets the segmentation and the temporal depth objective shape each other. The two-sentence version is this: use the lesion mask to decide where temporal depth smoothing is allowed, and use a surface crease, rather than a depth jump, as the geometric signature of a low-contrast lesion boundary. If this works, it should move the trade-off between boundary sharpness and temporal stability rather than pick a point on it.
Core Claim
Our hypothesis is that the stability-versus-sharpness conflict reported by EndoStreamDepth arises because temporal regularization is applied uniformly across the image. Gating that regularization with a jointly predicted lesion boundary, and replacing it inside the boundary band with a term that preserves the crease, should give lower depth flicker and sharper, steadier masks than three alternatives: a frozen-depth cascade, a 2D video memory model, and a multi-task model with uniform temporal loss.
The second half of the claim separates this proposal from MoDOT. MoDOT couples occlusion boundaries with depth on single indoor images. Its boundary-depth loss penalizes depth that fails to jump across a predicted boundary. Sessile and flat polyps, the hard cases in ETIS, usually do not produce a depth jump, because they do not occlude anything behind them. What they do produce is a change in surface orientation where the bump meets the wall. We therefore move the geometric boundary constraint from depth steps to creases in surface orientation, and from single frames to streams. MoDOT supplies the method template, a joint boundary-and-depth loss. Its static, occlusion-based form is precisely what we change.
Research Program
The network uses a pretrained depth backbone of the kind EndoStreamDepth builds on, the Depth Anything v2 ViT-L encoder. Two light decoders sit on top of it: a depth decoder and a mask decoder. Following EndoStreamDepth, each decoder carries streaming state-space modules at several scales. These modules, of the Mamba type, are recurrent layers that carry a compact hidden state forward one frame at a time at linear cost. A small cross-attention link lets mask features enter the depth decoder and depth features enter the mask decoder, in the spirit of MoDOT's cross-task module. A surface-normal map is computed from the predicted depth by finite differences. It needs no extra head.
The temporal term is the main ingredient. Let $D_t$ be the predicted depth at frame $t$. Let $\hat D_{t-1\to t}$ be the previous depth warped into frame $t$ using predicted relative pose and depth. Let $b_t(p) \in [0,1]$ be a soft band around the boundary of the predicted mask $M_t$, obtained by blurring the gradient of the mask. The gated temporal loss is
$$ \mathcal{L}_{\text{temp}} = \sum_p \big(1 - b_t(p)\big)\,\Big|\big(\log D_t(p) - \mu_t\big) - \big(\log \hat D_{t-1\to t}(p) - \hat\mu_{t}\big)\Big|, $$
where $\mu_t$ and $\hat\mu_t$ are the median log-depths of each map. Subtracting the medians makes the loss insensitive to a global change of scale, as in EndoStreamDepth's scale-invariant temporal loss. Inside the band we add a crease term:
$$ \mathcal{L}_{\text{crease}} = \sum_p b_t(p)\,\Big[\max\!\big(0,\ \tau - \|\nabla n_t(p)\|\big) + \big|\,\|\nabla n_t(p)\| - \|\nabla \hat n_{t-1\to t}(p)\|\,\big|\Big]. $$
Here $n_t$ is the normal map and $\tau$ is a small threshold. The first part asks for some change of surface orientation at the lesion margin. The second part asks that the strength of that change persist between frames. A low $\tau$, and a weight on the term that the mask confidence scales, keep the network from inventing creases on truly flat lesions. Because the gradients flow both ways, the mask can learn to sit on creases and the depth can learn to keep them.
Training data is the hard part, because no dataset in the file pairs depth ground truth with polyp masks. We train across datasets. Depth and pose supervision come from C3VD and the synthetic SimCol3D colons (SimCol3D), which have no masks. Mask supervision comes from SUN-SEG (Video Polyp Segmentation: A Deep Learning Perspective), which has no depth. On SUN-SEG clips, depth is anchored by pseudo-labels from a C3VD-fine-tuned teacher, and the gated temporal loss and the crease loss are applied without labels. Warping only needs windows of two to three frames. This limits exposure to the pose drift that the SimCol3D challenge documents on real video. Following PPSNet, shading-based supervision is attractive. However, its assumption that the tissue has uniform color fails on vascularized lesions, so if we use it at all, we restrict it to the predicted non-lesion region. The deployment target is at least 24 FPS, the streaming speed EndoStreamDepth reports. Video depth models that process whole clips at once, such as the generative-prior model in DVD, are out of scope for real-time use.
Baselines
The naive reference point is Depth Anything v2 run frame by frame for depth, together with a single-frame polyp segmenter. On C3VD, EndoStreamDepth reports that Depth Anything v2 and FlashDepth reach $\delta_1$ in the 0.730–0.850 range. Our model must exceed this by a wide margin, or the joint machinery is pointless.
For depth, the strongest competitor is EndoStreamDepth itself, at $\delta_1 = 0.952$ on C3VD. It is the strongest because it already solves streaming temporal stability with the same backbone family and the same kind of state-space decoder. We retrain it on our exact depth data with the same compute and tuning budget. We count our method as beating it if depth accuracy matches within noise and depth flicker on C3VD is lower, with flicker measured as the frame-to-frame standard deviation that EndoStreamDepth reports. The extra mask head must not cost depth accuracy.
For segmentation, the strongest video competitor is HFSTI-Net. It combines spatial features with Fourier-domain features and keeps a recurrent memory of past masks, reporting 86.27% Dice on SUN-SEG Hard at 31.27 FPS. We also run VPSentry, which keeps a long-term prototype memory and gates its updates when the scene changes abruptly, reporting 83.95% Dice on SUN-SEG Hard with 10% of labels. For single-frame segmentation, the strongest competitors are Polyp-DAM and Polyper. Polyp-DAM uses frozen depth input and reports 0.891 mean Dice on ETIS. Polyper refines the uncertain margin of a coarse mask with attention from the lesion interior, a margin it extracts with fixed erosion and dilation, and reports 86.51% mDice on ETIS. All baselines are retrained on the same SUN-SEG training split with matched augmentation. Beating them means higher Dice on SUN-SEG Hard and on the "Indefinable Boundaries" category, together with lower mask flicker.
The fairest constructed baseline is HFSTI-Net with frozen Depth Anything depth added as an extra input, fused the way Polyp-DAM fuses it. This is the obvious combination of the two existing ideas, and it is the one our method most needs to beat.
Two ablations isolate the key ingredient. The first keeps our architecture but replaces the gated temporal loss with a uniform one, setting $b_t \equiv 0$. The second swaps the crease term for MoDOT's depth-step constraint, or removes it entirely.
Why not just…?
Why not just feed frozen foundation depth into a strong video segmenter? This is the most serious threat, and the cascade baseline above tests it directly. The evidence against it is indirect. Frame-by-frame Depth Anything v2 flickers and reaches only 0.730–0.850 $\delta_1$ on C3VD (EndoStreamDepth). In a cascade, the segmenter cannot correct depth that specular highlights have corrupted, because no gradient reaches the frozen depth model. If the cascade matches our Dice and mask flicker on SUN-SEG Hard, we will conclude that joint training is unnecessary and report that.
Why not just smooth the outputs after inference? Moving averages or warp-and-blend filters do not change the features the encoder computes. A margin that collapses in one frame is therefore carried forward rather than fixed, which is the failure the HFSTI-Net and VPSentry papers describe. Such filters also lag during fast camera motion. We will include an exponential-moving-average filter on both the mask and the depth outputs, and test it on SUN-SEG's "Fast-motion" category, where the lag should show.
Why not just use 2D video memory without any geometry? HFSTI-Net is already strong, and geometry may add little on average. Our claim is narrower. Geometry should help where chromatic contrast is near zero. Polyp-DAM's 5.5-point Dice drop without depth is evidence for this, and HFSTI-Net itself reports remaining confusion from specular spots. We therefore stratify results by the "Indefinable Boundaries" category and by ETIS flat lesions. If there is no gain on these subsets, the geometric premise fails.
Why not just train a shared backbone with two heads and a uniform temporal loss? This is exactly our first ablation. MoDOT shows that naive two-head sharing underperforms cross-task coupling, with its cross-attention module improving RMSE by 7.1%, but only on static images. Whether gating matters in video is the open question this ablation answers.
Proposed Experiments
The decisive experiment is a trade-off curve. For each method, we sweep the weight of the temporal loss, or the strength of the smoothing filter for the post-hoc baseline. At each setting we plot boundary sharpness against temporal flicker. Depth sharpness is measured as the alignment of depth-gradient edges with ground-truth edges on C3VD. Mask sharpness is boundary F-score on SUN-SEG. Flicker is the frame-to-frame standard deviation of depth on C3VD and the warped consecutive-frame IoU of masks on SUN-SEG. The proposal succeeds only if the gated model's curve lies strictly beyond the uniform-loss ablation's curve, not merely at one better point on it.
Secondary checks follow. We report SUN-SEG Hard and Easy Dice against HFSTI-Net, VPSentry, and the reference point that the PNS+ paper reports (0.756 Dice on SUN-SEG Easy). We report ETIS and Kvasir-SEG Dice against Polyp-DAM and Polyper, with the crease-versus-depth-step ablation reported separately for flat lesions. We report depth on SimCol3D. For our depth model on that benchmark, the strongest available reference is RMSE 0.505 on Simcol-III from Self-supervised Monocular Depth and Pose Estimation for Endoscopy with Latent Priors. We found no reported result on Lacuna for EndoStreamDepth on SimCol3D. Finally, we report throughput on a single GPU against the 24 FPS target.
Two things are not measured anywhere in the file: mask flicker for any baseline, and depth accuracy on SUN-SEG. We will compute the first ourselves for every baseline. For the second, we will report qualitative depth results only.
References
- EndoStreamDepth: Temporally Consistent Monocular Depth Estimation for Endoscopic Video Streams
- Polyp-DAM: Polyp Segmentation via Depth Anything Model
- Occlusion Boundary and Depth: Mutual Enhancement via Multi-Task Learning
- SimCol3D - 3D reconstruction during colonoscopy challenge
- Video Polyp Segmentation: A Deep Learning Perspective
- Leveraging Near-Field Lighting for Monocular Depth Estimation from Endoscopy Videos
- DVD: Deterministic Video Depth Estimation with Generative Priors
- HFSTI-Net: Hierarchical Frequency-spatial-temporal Interactions for Video Polyp Segmentation
- VPSentry: Semi-supervised Video Polyp Segmentation via Sentry-guided Long-term Prototype Fusion with Correlation Dynamic Propagation
- Polyper: Boundary Sensitive Polyp Segmentation
- Self-supervised Monocular Depth and Pose Estimation for Endoscopy with Latent Priors