During endoscopic surgery the camera's view is repeatedly spoiled by things that are not anatomy. Water droplets and debris sit on the lens, smoke drifts through the field, and specular glare flashes off wet tissue. At the same time the tissue itself breathes, pulses, and stretches under the surgeon's instruments. Current 3D reconstruction methods handle one of these problems but not both. Some ignore the view degradation and treat droplets as geometry; others model the degradation but assume the organ stays still. This note proposes a reconstruction method that tells the two apart using a simple physical cue. Dirt on the lens moves with the camera, so it stays fixed in image coordinates. Tissue moves with the world and with its own deformation, and an independent point tracker can follow that motion. The same separation also gives a per-pixel map of how far the reconstruction can be trusted, which we test as a way to catch silent failures without ground-truth labels.
The opportunity matters because the two capabilities have so far lived in different systems. EndoMD-SLAM splits each colonoscopy frame into a persistent static field and a per-view transient field, and cuts trajectory error to 3.02 mm on degraded C3VDv2 sequences. It also assumes rigid or semi-rigid anatomy and runs at 0.95 FPS. Endo-4DGS and HFGS reconstruct deforming tissue at 70–100 FPS, but both depend on clean photometric consistency across frames. The obvious fix is to bolt a transient field onto a deformation field. That runs into a real ambiguity: a flexible enough deformation model can "explain" a droplet as a bulge in the tissue, and a flexible enough transient field can "explain" real tissue motion as noise. This ambiguity is the problem the proposal addresses.
Core Claim
Transient optical interference and non-rigid tissue deformation can be separated in monocular endoscopic video if each is tied to its own reference frame. The transient layer is anchored to the camera and must vary slowly in image space. The tissue layer is anchored to deforming 3D Gaussians and must agree with 2D tracks from an external causal tracker. Neither layer is allowed to explain the other's motion. We borrow the framing of "matrix interference" from analytical chemistry, where background material in a real sample distorts the signal of interest. In that framing, the lens-and-air layer is the background matrix and the tissue is the analyte. The motion-frame cue plays the role that standard addition plays in chemistry: an independent reference that lets the two be pulled apart without destroying the sample.
We model each frame $I_t$ as
$$I_t(u) = T_t(u) + \big(1 - \alpha_t(u)\big)\, R\big(G, D_t, P_t\big)(u).$$
In this equation, $u$ is a pixel, and $T_t$ and $\alpha_t$ are the transient colour and opacity. Both are predicted by a small convolutional network, a network that applies the same local filters across the whole image, operating in camera coordinates. $R$ renders the canonical Gaussians $G$ under the deformation $D_t$ and camera pose $P_t$. The compositing form is taken from EndoMD-SLAM. What is new is how each term is constrained. $T_t$ and $\alpha_t$ pay a penalty for changing between consecutive frames in image coordinates, which is cheap for lens-bound contaminants and expensive for anything that moves with the scene. Projected Gaussian anchors under $D_t$ must follow the 2D tracks produced by LiteTracker. LiteTracker is a causal point tracker that reaches 97.45% occlusion accuracy on SuPer at 29.67 ms for 1,024 points. It is a separate model and does not see the reconstruction, so its tracks are a second opinion about tissue motion.
The claim has a clear scope. When the camera is moving relative to the tissue, lens-bound interference and tissue motion have different image-space signatures, and the cue applies. Diffuse smoke is not lens-bound, but it is low-frequency and slow in image space, so the transient prior still roughly fits it. Blood pooling on the tissue surface is a real change in tissue appearance, not interference. We handle it with a time-varying colour term on the Gaussians and do not claim to remove it. When the camera is stationary, the cue disappears, and we report that regime separately instead of hiding it.
Research Program
The system has three parts. The first is a deformable Gaussian backbone based on Endo-4DGS. It keeps that paper's Depth-Anything pseudo-depth initialisation and confidence-guided depth regularisation, and it adds HFGS's optical-flow temporal loss as an option. The second is a camera-anchored transient network. This is a lightweight U-Net-style convolutional network. We choose a CNN because OpenSRH found that convolutional inductive biases train more stably than vision transformers on limited medical texture data, and because PrecisionView shows that a modified U-Net can undo optical degradation in real time for endomicroscopy. The third is the track-consistency term fed by LiteTracker running causally alongside the reconstruction. A reliability gate like EndoMD-SLAM's pauses updates to $G$ when the average $\alpha_t$ passes a threshold. We refine that gate to act per region instead of per frame, so tissue that is still visible keeps updating.
The fourth element is the validation lens. The clinical validation literature separates technical validation on retrospective benchmarks from operational validation, which covers latency, distribution shift, and behaviour in workflow. Real deployments have no labels, so they need signals that flag degraded outputs. Label-free estimation under distribution shifts builds such signals from classifier confidence, reaching about 0.05 mean absolute error out of distribution on chest radiographs. It fails under prevalence shift and has never been tested on video. We ask whether the separated opacity map $\alpha_t$, which is a physically meaningful quantity, predicts where depth and tracking errors will occur better than confidence-style signals do. We also commit to operational targets: at least 20 FPS end to end on one RTX 3090-class GPU. We report latency honestly against EndoMD-SLAM's 0.95 FPS.
Baselines
The naive reference is Endo-4DGS run unchanged on degraded video. It is the clean-field deformable reconstruction method whose failure motivates the work, and it tells us how much the degradation costs. We run it with the same pseudo-depth, the same iteration budget, and the same resolution as our method.
The strongest existing method for degraded video is EndoMD-SLAM. On degraded C3VDv2 it reports an absolute trajectory error of 3.02 mm, against roughly 35 mm for EndoGSLAM, and a 9.9 dB PSNR gain over standard 3DGS. Because C3VDv2 is rigid colon phantom data, this is EndoMD-SLAM's home ground. Our method beats it only if it matches that error within noise on C3VDv2 while running at least ten times faster, and clearly beats it on deforming sequences, where its rigidity assumption should break. For clean deformable reconstruction, the strongest number in the file is HFGS at 38.14 PSNR on EndoNeRF. On clean EndoNeRF our method must stay within about 0.5 dB of HFGS, so that the added machinery costs nothing when there is no interference. Endo-4DGS reports beating earlier methods on StereoMIS, but the file quotes no PSNR or SSIM value for it. We found no reported result on Lacuna for Endo-4DGS on StereoMIS, so we will rerun it. For tracking, LiteTracker's 97.45% occlusion accuracy on SuPer is the reference. We will report how that figure degrades under our synthetic interference, with and without our transient layer used to reweight its inputs.
We also run three ablations to isolate the key ingredient. The first removes the image-space temporal penalty, leaving an unconstrained transient field plus deformation; this is the naive bolt-on. The second removes the track-consistency term. The third replaces the CNN transient predictor with EndoMD-SLAM's per-view optimised field. Every variant gets the same data, the same per-scene optimisation budget in wall-clock time, and hyperparameters tuned on one held-out sequence per dataset.
Why not just…?
Why not just segment the degraded pixels with a CNN and mask them out of the Endo-4DGS loss? This is the most serious alternative, and it may work well for opaque, sharp-edged debris. Two things count against it. It needs labelled degradation masks, which no dataset in the file provides for deforming tissue. It also treats partial attenuation, such as thin water films or haze, as all-or-nothing, so it either throws away tissue that is still visible or keeps corrupted colour. We include a supervised mask baseline trained on the same synthetic overlays our method sees. If it matches our method on semi-transparent interference, the compositing model is not needed, and we will say so.
Why not just rely on robust photometric losses such as HFGS's Charbonnier and Census terms? Robust losses down-weight outliers but do not stop the deformation field from absorbing them. A droplet that persists over many frames is not an outlier; it is a consistent signal that the optimiser will fit with geometry. Running HFGS on the degraded composites tests this directly, and we expect errors to concentrate on persistent contaminants.
Why not just gate out bad frames, as EndoMD-SLAM does, or have staff flush the lens? Flushing is itself interference. EndoMD-SLAM shows that standard systems treat flushed water as anatomy and drift to about 35 mm error. Frame-level gating throws away whole frames during exactly the moments when surgeons act on the tissue. Our region-level gate ablation measures how much tissue coverage and accuracy frame-level gating gives up.
Why not just use sparse rigid tracking such as ORB-SLAM3? On rigid C3VD it matches dense methods, with 0.32 mm error according to EndoGSLAM. However, it produces no surface to overlay guidance on, and it breaks under the centimetre-scale displacements reported in liver registration work (Latent-grounded liver registration). It is not a competitor on deforming scenes.
Proposed Experiments
No existing benchmark contains both deformation and labelled optical degradation. Building one is the first contribution, and its limits need to be stated plainly. We will overlay physically motivated interference onto EndoNeRF and StereoMIS: droplets fixed to the lens with refraction, low-frequency smoke haze, and specular bursts. For SuPer and STIR we will add the same overlays but apply the lens layer to all frames with consistent camera-fixed placement. Because the overlays are synthetic, the clean frames serve as exact ground truth for both layers. The unmodified degraded C3VDv2 sequences provide real interference on rigid geometry. We will also collect a small number of real clinical clips with naturally occurring fouling, which have no ground truth and will be assessed only by blinded surgeon ratings of overlay stability.
We report five metrics: PSNR and SSIM of the recovered clean tissue render against clean frames; depth error against SCARED ground truth where available; trajectory error on C3VDv2; tracking accuracy on SuPer and STIR; and intersection-over-union of thresholded $\alpha_t$ against the known overlay masks.
The decisive check for the core claim is a sweep over camera motion magnitude. If motion-frame anchoring is what separates the layers, separation quality should fall toward the naive bolt-on ablation as camera motion approaches zero, and it should rise with motion. If separation quality stays flat, the gain comes from somewhere else and the claim is wrong.
The second decisive check concerns trust. For each frame we flag pixels whose depth or track error exceeds a clinically motivated threshold of 2 mm. We then compare AUROC at predicting those pixels for three signals: our opacity $\alpha_t$, the raw photometric residual, and an ATC-style thresholded confidence adapted from the label-free estimation work. We also simulate a prevalence-like shift by varying the fraction of heavily fouled frames, which is the condition where that paper's estimators failed. The opacity map has an advantage only if it beats both alternatives and stays calibrated across that shift. The last check is the operational bar: end-to-end throughput of at least 20 FPS with LiteTracker running alongside, reported with latency percentiles, not just a mean.
References
- EndoMD-SLAM: Endoscopic Gaussian Splatting SLAM under Optical Degradation with Memory and Static-Transient Decomposition
- Endo-4DGS: Endoscopic Monocular Scene Reconstruction with 4D Gaussian Splatting
- HFGS: 4D Gaussian Splatting with Emphasis on Spatial and Temporal High-Frequency Components for Endoscopic Scene Reconstruction
- LiteTracker: Leveraging Temporal Causality for Accurate Low-Latency Tissue Tracking
- OpenSRH: optimizing brain tumor surgery using intraoperative stimulated Raman histology
- Dual-modality, deep-learning-enabled endomicroscope with large field-of-view and depth-of-field for real-time in vivo imaging of epithelial hallmarks of cancer
- Label-free estimation of clinically relevant performance metrics under distribution shifts
- EndoGSLAM: Real-Time Dense Reconstruction and Tracking in Endoscopic Surgeries Using Gaussian Splatting
- Preoperative-to-intraoperative Liver Registration for Laparoscopic Surgery via Latent-Grounded Correspondence Constraints