Many boundaries between soft tissues are nearly invisible in a single endoscopic frame. Adjacent tissues share color and texture, and wet, specular surfaces wash out what little edge there is. Over the course of surgery, though, these tissues often move differently. A grasper pulls one structure while its neighbor slides or stays put, and breathing shifts organs by different amounts. This proposal treats that difference in motion as the missing contrast. We reconstruct the deforming surgical scene as a cloud of moving 3D Gaussians, which are small colored blobs that are rasterized to render images. Each Gaussian carries a full motion trajectory across the video. We then require the semantic labels attached to the Gaussians to change wherever those trajectories diverge. The approach depends on the trajectories being reliable over long videos and behind surgical tools, so we drive them with a linear-time sequence model rather than the local interpolators current methods use.
In two sentences: tissues that look alike can still be separated if they move differently, so a 4D reconstruction should carry long-horizon motion and turn motion discontinuities into a boundary signal for segmentation. Doing this at the scale of hundreds of thousands of Gaussians over hundreds of frames requires a global temporal model whose cost grows linearly. That model also has to carry each Gaussian's state across stretches where an instrument hides the tissue.
What exists and where it stops
Surgical 4D Gaussian splatting is now fast and accurate at rendering. EndoGS deforms Gaussians with a small network reading HexPlane features, a factorization of space-time into six 2D feature grids introduced for surgery by Forplane. EndoGaussian and Endo-4DGS add monocular depth priors for initialization. These papers supply our reconstruction machinery. They also constrain the design. Each deformation is a local lookup at $(x, t)$, so nothing ties a Gaussian's position at one time to its history. EndoGS reports that geometry collapses behind instruments unless heuristic masks are used.
The closest work is Feature-EndoGaussian. It distills features from the Segment Anything Model (SAM) into the Gaussians so that labels deform with tissue. It reaches 0.93 Dice on binary EndoVis18 tasks but only 0.77 on multi-label tissue segmentation, and that gap motivates this proposal. Its labels come from appearance features alone. Once the Gaussians are fit, it never uses the fact that each one has a trajectory. HFGS attacks "melted" tissue edges with frequency-domain and optical-flow losses, but it improves rendering fidelity, not semantic delineation.
Two lines of work bring state space models into this space. A state space model is a recurrent sequence model whose cost is linear in sequence length; the selective variant Mamba makes its update depend on the input. Vivim uses forward, backward and spatial Mamba scans to segment blurry ultrasound lesions in 2D video. Its boundary term is a 2D affine regularizer with no 3D geometry. Gaussians on their Way treats Gaussian motion as a state-space system and adds a Kalman-style consistency filter, evaluated on synthetic D-NeRF scenes. It smooths motion but never reads motion as a segmentation cue. Static Mamba segmenters for low-contrast organs, such as DB-MSMUNet and DeMambaNet, show that global context helps on fuzzy margins. Both work on single 2D slices.
None of these works turns relative motion into a boundary signal. That is the claim we test.
Core Claim
On low-contrast surgical tissue, a large share of the true boundaries coincide with discontinuities in long-horizon 3D motion. Appearance alone cannot recover these boundaries. A 4D reconstruction can recover them, provided its trajectories stay trustworthy through occlusion. We expect the gain to concentrate in two places: boundary pixels where appearance contrast is lowest, and regions that are occluded during part of the video.
Research Program
The reconstruction starts from a standard surgical 4DGS pipeline, following EndoGaussian-style initialization and a HexPlane deformation. We add a trajectory model on top. For each Gaussian, the HexPlane features at its canonical position, sampled at every frame, form a sequence of length $T$. A visibility flag is appended at each frame, computed from instrument masks and rendered opacity.
The trajectory model has two parts. A bidirectional selective scan runs along time for each Gaussian and outputs a deformation residual. Because the scan is selective, it can learn to ignore frames where the Gaussian is hidden and to carry its state across those frames from both sides. A second scan runs over Gaussians ordered along a space-filling curve in canonical space, so distant tissue regions can share context, for example a common breathing phase. Total cost is $O(GT)$ for $G$ Gaussians and $T$ frames. The final hidden state of the temporal scan becomes a compact motion descriptor $z_i$ for Gaussian $i$.
Each Gaussian also carries a semantic feature $f_i$, distilled from SAM as in Feature-EndoGaussian and supervised by rendered masks. For neighboring pairs $(i,j)$ in canonical space we add a motion-contrast term:
$$ \mathcal{L}_{mc} = \sum_{(i,j)} w_{ij}\,\|f_i - f_j\|^2 + (1 - w_{ij})\,\max\!\big(0,\, m - \|f_i - f_j\|\big)^2, \qquad w_{ij} = \exp\!\big(-\|z_i - z_j\|^2/\sigma^2\big). $$
Here $w_{ij}$ measures how similarly the two Gaussians move, $m$ is a margin, and $\sigma$ is a bandwidth. Pairs that move alike are pulled toward the same label, and pairs that move differently are pushed apart. Many tissues are mechanically coupled, so most boundaries carry no motion contrast at all. To avoid harming those regions, we scale the term by appearance uncertainty, estimated from the entropy of the distilled SAM prediction. Where appearance is confident, the term contributes little.
Analysis is as much a part of the program as the method. Before any training, we measure from fitted trajectories what fraction of annotated tissue-boundary pixels on EndoVis18 show a motion discontinuity above noise. If that fraction is small, the idea fails, and we would report this as a negative finding.
Baselines
The naive reference point is Feature-EndoGaussian run unchanged. Its reported numbers on EndoVis18 are 0.93 Dice for binary tasks and 0.77 Dice for multi-label tasks, and it reports 39.1 PSNR on EndoNeRF. It is also the strongest existing method on this exact question, because it is the only cited system that produces deforming 3D semantic labels on surgical video. We will rerun it with its released settings on our hardware. Every variant below will use the same Gaussian budget, iteration count and per-scene optimization time cap, with hyperparameters for each searched over the same small grid. Beating it means a multi-label Dice gain that holds across scenes, and a larger gain in boundary metrics, namely boundary F-score and 95th-percentile Hausdorff distance, on low-appearance-contrast and occluded strata. No boundary-metric numbers for it were found on Lacuna, so we will compute them ourselves.
The second baseline isolates the loss question. It is Feature-EndoGaussian plus an explicit contour loss on rendered masks, in the style of the contour-loss segmentation work of Niu et al.. This tests whether Direction 3's standard medicine, boundary supervision without motion, already closes the gap.
The third baseline is Vivim trained as a 2D video segmenter on the same EndoVis18 frames. It is the strongest linear-time model for segmenting ambiguous boundaries in video. Its reported 0.8324 Dice is on thyroid ultrasound (VTUS). No reported result for Vivim on EndoVis18 was found on Lacuna.
Three ablations isolate the key ingredients. The first keeps the motion-contrast loss but replaces the selective scan with the Kalman-style consistency filter of Gaussians on their Way. That paper reports 34.45 PSNR on D-NeRF, and there is no reported surgical result for it on Lacuna. The second computes $w_{ij}$ directly from raw HexPlane trajectories with no sequence model. The third uses the full model but removes the motion-contrast loss, so the sequence model only affects geometry.
Why not just…?
Why not just add a boundary loss to Feature-EndoGaussian? The polyp literature, for instance the boundary-attention network of Kang et al., shows that boundary supervision helps when some edge evidence exists in the features. Our target cases are the ones where SAM's appearance features carry almost none. A contour loss there can only fit annotator preference. The second baseline settles the question directly. If it matches our method on the low-contrast stratum, motion adds nothing, and we would say so.
Why not just compute boundaries from raw trajectories, with no sequence model? This is the most serious threat. Per-Gaussian trajectories already exist in any 4DGS model, and comparing them between neighbors is cheap. The reason to expect it to fall short is occlusion. EndoGS documents that HexPlane deformations are unconstrained behind instruments, so raw trajectories there are arbitrary and would produce false boundaries exactly where tools pass. The second ablation, stratified by how long each region is occluded, will show whether this holds. If raw trajectories match the full model even in heavily occluded regions, we will drop the sequence model and present motion contrast alone as the contribution.
Why not just use 2D optical-flow discontinuities with a 2D segmenter? 2D flow edges mix together several things: camera motion, depth edges, and specular highlights sliding across wet tissue. HFGS already needs extra losses to make flow priors usable. We will add a variant that builds $w_{ij}$ from projected optical-flow discontinuities instead of 3D trajectories. Comparing it against our method on the specular and depth-edge strata will show whether lifting to 3D matters.
Proposed Experiments
The main benchmark is EndoVis18, using the multi-label tissue segmentation protocol of Feature-EndoGaussian. We report Dice, boundary F-score and HD95. Every test pixel is split into strata by local appearance contrast, measured as image gradient magnitude in a band around the ground-truth boundary, and by occlusion duration. Rendering quality must not drop as a side effect. We check this on EndoNeRF against the reported 39.1 PSNR (Feature-EndoGaussian), 38.14 (HFGS) and 37.849 (EndoGaussian), and on SCARED against the 27.3 PSNR reported by Feature-EndoGaussian. StereoMIS serves as a held-out stress test with longer sequences. We also record frames per second and memory as $T$ grows, to confirm linear scaling against a temporal attention variant.
The decisive checks are three. First, the pre-training measurement of how often boundaries carry motion contrast. Second, whether the Dice and boundary-metric gains concentrate in the low-contrast and occluded strata, as the claim predicts, rather than spreading evenly. Third, whether the full model beats the raw-trajectory ablation specifically in occluded regions. Failing the first check refutes the idea. Gains that appear only in high-contrast regions would mean the improvement comes from somewhere other than motion. Failing the third would cut the proposal down to the simpler motion-contrast loss without the sequence model.
One limitation is acknowledged in advance. The file contains no multi-annotator surgical dataset, so we cannot test whether motion-derived boundaries reduce disagreement between annotators. That question stays open for future work.
References
- EndoGS: Deformable Endoscopic Tissues Reconstruction with Gaussian Splatting
- Efficient Deformable Tissue Reconstruction via Orthogonal Neural Plane
- EndoGaussian: Gaussian Splatting for Deformable Surgical Scene Reconstruction
- Endo-4DGS: Endoscopic Monocular Scene Reconstruction with 4D Gaussian Splatting
- Feature-EndoGaussian: Feature Distilled Gaussian Splatting in Surgical Deformable Scene Reconstruction
- HFGS: 4D Gaussian Splatting with Emphasis on Spatial and Temporal High-Frequency Components for Endoscopic Scene Reconstruction
- Vivim: a Video Vision Mamba for Medical Video Object Segmentation
- Gaussians on their Way: Wasserstein-Constrained 4D Gaussian Splatting with State-Space Modeling
- DB-MSMUNet: Dual Branch Multi-Scale Mamba UNet for Pancreatic CT Scans Segmentation
- DeMambaNet: Deformable Convolution and Mamba Integration Network for High-Precision Segmentation of Ambiguously Defined Dental Radicular Boundaries
- A Medical Image Segmentation Method based on Multi-scale Features and Contour Loss Constrain (S)
- Multi-scale information sharing and selection network with boundary attention for polyp segmentation