Methods that reconstruct moving, deforming soft tissue from endoscope video now render at 70–100 frames per second and train in minutes. Their reported quality, though, comes almost entirely from a handful of short clips chosen because the tissue is visible, the lens is clean and the camera moves slowly. Clinical machine learning has seen this pattern before. Models validated on curated historical records lose accuracy when they run live on every patient. One prospective study showed that much of that loss came from differences in the data pipeline, not from the patients (Mind the Performance Gap). Nobody has measured the equivalent gap for deformable surgical vision.
We propose to measure it and take it apart. The study has two crossed factors. The first is clip selection: curated clips versus consecutive, unselected segments of full-length surgical video. The second is processing mode: offline optimization over the whole clip versus causal streaming, where each frame is processed using only past frames, automatic tool masks are used, and the system runs at the live frame rate. Crossing the two separates the drop caused by curation from the drop caused by the "infrastructure" of live processing. On top of this decomposition we test whether a cheap correction can predict live performance without a live trial. The correction reweights per-condition benchmark results (smoke, fluid on the lens, tool occlusion, fast motion) by how often each condition actually occurs in surgery. We check that prediction against a silent-mode capture: the system runs alongside consecutive operations without being shown to the surgeon.
Core Claim
Our hypothesis is that curation and offline processing together, rather than patient anatomy or pathology, account for most of the gap between benchmark and live quality for current 4D reconstruction methods. Here a 4D model means a 3D scene that changes over time. If the hypothesis holds, a reweighting of retrospective results by condition prevalence should predict silent-mode performance far better than the raw benchmark number does. If reweighting fails, the residual tells us something measurable: there are failure modes in live surgery that no curated condition captures, which is exactly the "high accuracy mirage" argument applied to surgical vision.
The idea is distinct from building another robust reconstruction method. EndoMD-SLAM (EndoMD-SLAM) is the closest work. It showed that lens contamination raises tracking error for the rigid EndoGSLAM pipeline from about 0.34 mm to about 35 mm, and that its own gating brings this down to 3.02 mm. That study has three limits relative to ours. It used a benchmark filtered to oversample degradation, so the sample is curated in the opposite direction and says nothing about how often degradation actually occurs. It assumes rigid scenes. It never separates selection effects from processing-mode effects. The P2ILF challenge (MICCAI2022 P2ILF) concluded that no registration method was clinically ready. However, it tested on 16 frames from 2 patients, selected for landmark visibility, so it cannot say why the methods fail or how often. Our contribution is to adapt the Ötles decomposition to deformable video, which neither line of work has done.
Research Program
Ötles et al. split the live drop of a tabular infection model into two parts. A data-pipeline component cost 0.016 in AUROC (area under the ROC curve). A temporal component gained 0.005 (0.778 retrospective versus 0.767 prospective). They did this by comparing three snapshots of data. Our version replaces their snapshots with a two-by-two design.
For a quality metric $M$, arm $A$ is curated clips processed offline, as in the published benchmarks. Arm $B$ is the same curated clips processed causally in streaming mode. Arm $C$ is consecutive unselected segments processed causally, which is what a live system would face. Arm $D$ is consecutive segments processed offline. The total gap splits as
$$ M_A - M_C = \underbrace{(M_A - M_B)}_{\text{processing mode}} + \underbrace{(M_B - M_C)}_{\text{selection}}, $$
and arm $D$ lets us estimate the interaction $(M_A - M_B) - (M_D - M_C)$. The interaction answers a practical question: does streaming hurt more on hard footage than on easy footage?
The processing-mode factor is concrete. Forplane (Efficient Deformable Tissue Reconstruction via Orthogonal Neural Plane) and the 4D Gaussian splatting methods Endo-4DGS (Endo-4DGS) and HFGS (HFGS) all optimize per clip. They see future frames, and most use given tool masks. Gaussian splatting represents the scene as many small semi-transparent ellipsoids projected onto the image. In streaming mode, each method is updated incrementally within a fixed time budget per frame. It sees only frames up to time $t$, uses automatically generated tool masks, and receives frames through the actual capture path (frame grabber, compression, timing jitter). We also log outright failures: divergence, missed deadlines and crashes. Ötles et al. found their live pipeline dead on 10 of 356 days, and this outage rate is the analogue.
Evaluation without ground-truth depth needs care. Our primary metric is one-step-ahead rendering: the model, updated through frame $t-1$, renders frame $t$, and we score it with PSNR, SSIM and LPIPS against the observed frame. The offline arms are scored on the same held-out frames for comparability. Photometric scores are partly circular on degraded frames, because the target itself is fogged. We therefore add a geometric check. Surgeons annotate sparse anatomical landmarks on frames sampled uniformly at random, not frames chosen for visibility, and we measure 2D landmark tracking error and the fraction of landmarks the system loses.
Every segment is also tagged for six conditions: smoke, fluid or blood on the lens, flushing, tool occlusion, fast camera motion and active cutting. These tags feed the reweighting estimator. It predicts live quality as $\hat M_C = \sum_k \pi_k \, \bar M_{B,k}$, where $\pi_k$ is the prevalence of condition stratum $k$ in consecutive video and $\bar M_{B,k}$ is the mean streaming quality on curated-and-degradation-benchmark segments in that stratum. The silent-mode capture at a partner site then provides an observed $M_C$ on new consecutive cases to compare against.
For registration of preoperative models to live video, we run a smaller parallel arm. Land-Reg (Land-Reg) reaches 75.38% Dice and a mean target registration error (TRE) of 76.46 pixels on P2ILF. We evaluate it on P2ILF frames versus uniformly sampled frames from full liver-resection recordings, keeping the same annotation protocol. This tests whether the selection effect carries over from reconstruction to registration.
Baselines
The naive reference is the published retrospective number taken at face value, which is what a reader of the benchmarks would assume carries over to surgery. On the in-vivo EndoNeRF clips from robotic prostatectomies, HFGS reports PSNR 38.14 and Forplane reports PSNR 37.31. Our arm $A$ reproduces these. A reproduction within about 1 dB is a precondition, so that any gap we report cannot be blamed on a weak reimplementation.
The strongest existing reconstruction methods are HFGS and Endo-4DGS. They are the fastest, highest-quality deformable methods in the file, and therefore the most plausible candidates for live use. Both are run in all four arms with the authors' settings. They get the same GPU (an RTX 3090-class card, matching the HFGS report) and the same per-frame time budget in streaming mode. Each also gets an equal, small tuning budget for its incremental-update schedule, so streaming is not handicapped by naive defaults.
EndoMD-SLAM is the strongest degradation-robust competitor despite assuming a rigid scene. It reports 3.02 mm tracking error on C3VDv2 at 0.95 FPS. We run it in arms $B$ and $C$ to test whether handling lens contamination without modelling deformation shrinks the selection gap more than handling deformation without modelling contamination does.
For the estimator, the comparisons are the naive estimate $\hat M_C = M_A$ and an unweighted average over the degradation benchmark. "Beating" these means a smaller absolute prediction error against observed silent-mode $M_C$, with bootstrap confidence intervals over cases.
The key ablation drops arm $B$. Collapsing to a two-arm curated-versus-consecutive comparison folds processing mode into selection. If this changes which methods rank best, the decomposition earns its place.
Why not just…?
Why not just run a full prospective trial with surgeons viewing the overlay? That is the eventual standard of evidence. It is expensive and ethically heavy, and it yields one confounded number. Silent mode captures the same consecutive, unselected input stream at no risk to patients, and the factorial design says why performance drops. A trial cannot do this cleanly, because surgeon behaviour changes once guidance is shown.
Why not just evaluate on longer clips or the existing degradation benchmark? Longer clips remove only part of the curation, since the operator still chooses which surgery and which stretch of it. The C3VDv2 degradation subset deliberately oversamples contamination, so it has no information about how often contamination occurs. It is also rigid colonoscopy, not deforming abdominal tissue. Both are inputs to our estimator, not substitutes for consecutive sampling.
Why not just stratify by condition and skip the live capture? This is a real threat, and it is exactly our reweighting estimator. If reweighting predicts silent-mode quality within its confidence interval, the honest conclusion is that cheap stratified retrospective reporting is sufficient, and we would recommend that the community report it. The live capture is what tells us whether this is true. Without it, the stratified estimate is unverified.
Why not just fall back to rigid registration with surgeon landmarks? Feuerstein et al. (Automatic Patient Registration) obtained 2.6–3.2 mm only on a rigid phantom, and they noted that breathing and insufflation break the rigid assumption. Land-Reg's move from rigid to non-rigid registration raised Dice from 69.21% to 75.38%. We include rigid Land-Reg in the registration arm, so the comparison is measured under unselected frames rather than assumed.
Proposed Experiments and Decisive Checks
Data comes from three sources. The first is curated public clips: EndoNeRF, StereoMIS porcine sequences, Hamlyn and SCARED. The second is full-length recordings from which consecutive segments are drawn on a fixed time grid, with no exclusion except camera-out-of-body intervals, and with every exclusion logged. The third is a silent-mode capture of consecutive cases at one partner site.
We found no reported result on Lacuna for any deformable method on unselected surgical video, for streaming-mode PSNR on EndoNeRF, or for absolute PSNR on Hamlyn, StereoMIS or SCARED. These are measured here for the first time.
The first decisive check is the sign and size of the two components for HFGS and Endo-4DGS. Selection dominating processing mode, or the reverse, determines whether the field should invest in robustness or in causal architectures.
The second decisive check is the interaction term. A large positive interaction would mean offline benchmarks understate the difficulty of streaming on exactly the footage that matters.
The third decisive check is the estimator test. The reweighted prediction must beat the naive benchmark number on observed silent-mode quality, landmark loss rate and outage rate. If it does not, the unexplained residual, broken down by case attributes such as liver cirrhosis noted in P2ILF, becomes the next target. A residual that tracks pathology rather than optics would revive the concern raised by the Neural-Augmented Kelvinlet work (Neural-Augmented Kelvinlet): its uniform-stiffness, quasi-static assumption, and the local-isometry assumption in Land-Reg, break on diseased tissue.
References
- Mind the Performance Gap: Examining Dataset Shift During Prospective Validation
- EndoMD-SLAM: Endoscopic Gaussian Splatting SLAM under Optical Degradation with Memory and Static-Transient Decomposition
- An objective comparison of methods for augmented reality in laparoscopic liver resection by preoperative-to-intraoperative image fusion from the MICCAI2022 challenge
- Efficient Deformable Tissue Reconstruction via Orthogonal Neural Plane
- Endo-4DGS: Endoscopic Monocular Scene Reconstruction with 4D Gaussian Splatting
- HFGS: 4D Gaussian Splatting with Emphasis on Spatial and Temporal High-Frequency Components for Endoscopic Scene Reconstruction
- Preoperative-to-intraoperative Liver Registration for Laparoscopic Surgery via Latent-Grounded Correspondence Constraints
- Automatic Patient Registration for Port Placement in Minimally Invasixe Endoscopic Surgery
- Neural-Augmented Kelvinlet: Real-Time Soft Tissue Deformation with Multiple Graspers