Surgeons working through a keyhole see only a narrow, often blocked slice of the anatomy. Computer reconstructions that rebuild the surgical scene in 3D could restore the missing context, but the best current methods must fit a fresh model to every short video clip. That fitting takes minutes. A real operation lasts hours, and its scene changes continuously as the camera moves, tools push tissue, and the surgeon cuts. We propose to keep one persistent 3D model of the scene for the whole procedure. Each time a new few-second window of video arrives, a single trained network predicts how that model should change: which pieces stay as they were, which move, and which must be replaced. The network does this in one pass, with no optimization loop. It also outputs a "freshness map" recording how recently each region was confirmed by the camera, so that an overlay can tell the surgeon which parts of the reconstruction may be out of date.
The idea borrows its core move from longitudinal radiology, where the strongest systems do not re-read each scan from scratch. They encode the current scan relative to a prior one and decide which earlier findings still hold. In that field, simply carrying the last scan forward is a baseline that many sophisticated models fail to beat. We expect the same to be true inside one operation: "reuse the previous reconstruction" will be hard to beat. The proposal is built around beating it, and around beating warm-started optimization on the trade-off between latency and quality.
Core Claim
The scene is represented with 3D Gaussian splatting, which describes it as a cloud of small, colored, semi-transparent ellipsoids that can be rendered very quickly. Let $G_{t}$ be this Gaussian scene at update step $t$ and let $W_{t+1}$ be the next short video window. We train a network $f$ that outputs $G_{t+1}$ in a single forward pass. For each existing Gaussian, $f$ produces a keep gate $g_i \in [0,1]$ and a displacement. It also produces a set of newly spawned Gaussians for surface that has just become visible, and a freshness value for every Gaussian. Our claim is that this gated update reaches reconstruction quality comparable to per-clip fitting on the current window, keeps quality on regions outside the window better than re-fitting from scratch, and runs in well under a second. That last property would place it on a latency–quality curve that warm-started optimization cannot reach at equal latency.
This differs from three existing lines of work.
The first is surgical per-scene reconstruction. EndoGaussian fits deformable Gaussians to a clip in under two minutes. Forplane factorizes space-time into six planes and fits in ten minutes. Both treat each clip as a self-contained episode with no carried state. They define the fitting cost we want to remove, and they also serve as our quality reference.
The second is feed-forward reconstruction from general video. DGS-LRM predicts deformable Gaussians and scene flow from monocular video in about 0.6 seconds. DepthSplat and GS-LRM do the same for posed multi-view images. These models rebuild every input from nothing. They were trained on natural or synthetic scenes, and DGS-LRM itself reports blur and floating artifacts away from the input cameras. They supply our backbone design, but none of them conditions on a previous scene state.
The third is longitudinal radiology, which supplies the mechanism. DeltaNet computes an explicit image difference against a prior exam and uses a gate to decide which earlier findings remain active. BioViL-T separates static from interval-change features. CRONOS shows that carrying the last scan forward is a baseline that ConvLSTM and SimVP fail to beat. That last finding constrains us: it obliges us to report the carry-forward baseline honestly. Our contribution is to move this gate-the-history idea from 2D reports into a 3D scene representation that updates every few seconds.
Research Program
The network has two inputs. The first is the new window, encoded with a DGS-LRM-style space-time transformer whose features are initialized from a monocular depth model, as EndoGaussian and DepthSplat do. The second is the current Gaussian scene, rendered into the new window's viewpoints as color, depth, and Gaussian-ID images. Rendering the old scene into the new views gives a direct image-space difference, the 3D analogue of DeltaNet's difference map.
Cross-attention between the two streams then predicts three things. Per-Gaussian outputs (the keep gate and the displacement) are read from the rendered-ID tokens. Spawned Gaussians are predicted per pixel wherever the rendered old scene leaves holes or disagrees strongly with the new frames.
Freshness is not a separately predicted quantity. It is reset to one for Gaussians the network confirms in view and decays otherwise. This makes it interpretable by construction, since it is a record of when each region was last seen.
Training uses only self-supervision from video, because no long annotated surgical sequences exist in the file. We cut each available sequence into a history segment and a later window. The history is fitted once with EndoGaussian to produce $G_t$, and the network is trained to update it. The loss has three parts: a photometric rendering loss on held-out frames of the later window; a "revisit" loss on frames from the history segment, showing regions absent from the new window, which penalizes destroying correct memory; and a small penalty on gate churn, which discourages needless replacement. We will also train on a curriculum of artificially long sequences made by chaining augmented windows, so the network sees many consecutive updates and we can measure drift.
Preoperative CT registration, as done by Self-P2IR and Land-Reg, is the natural first entry of this timeline. We leave it out of scope here, because the P2ILF data contain keyframes rather than continuous video.
Baselines
The naive reference point is carry-forward: keep $G_t$ unchanged and render it for the new window. This is the direct analogue of CRONOS's last-context-image baseline, which in that paper beats ConvLSTM and SimVP and loses to CRONOS only modestly on Lumiere (normalized root-mean-square error 7.55 versus 8.38). It costs nothing, preserves memory perfectly, and fails only where the tissue has actually changed. Beating it on current-window frames while matching it on revisit frames is the minimum bar.
The strongest existing method is per-clip EndoGaussian fitted from scratch on each new window. It reports a peak signal-to-noise ratio (PSNR) of 37.849 on ENDONERF, with fitting under two minutes and rendering at 195.09 frames per second. For comparison, Forplane reports 37.31 PSNR with ten-minute fitting, and the original EndoNeRF reports 29.48. Those numbers come from whole-clip protocols. We will re-run EndoGaussian with its released settings on exactly our windows, on the same GPU, so that both methods see identical frames. Beating it means coming within about 1 dB PSNR on current-window frames, at more than 50 times lower update latency, while exceeding it on revisit frames, which a from-scratch fit cannot represent at all.
The baseline most likely to compete is warm-started EndoGaussian. It initializes from $G_t$ and runs $k$ optimization iterations on the new window. We sweep $k$ so its wall-clock latency spans from our network's latency up to the full two minutes, giving it the same total compute budget we spent tuning our model.
We also run feed-forward without memory: our backbone fine-tuned on the same surgical data, but predicting $G_{t+1}$ from the window alone. There is no reported result found on Lacuna for feed-forward reconstructors such as DGS-LRM on ENDONERF, SCARED, or Hamlyn, so this baseline is new.
Two ablations isolate the key ingredient. One removes the rendered-history difference input. The other replaces the learned gate with "keep everything and only add."
Why not just…?
Why not just warm-start the per-scene optimizer from the previous state? This is the real threat. It inherits memory for free, and a few hundred iterations may suffice when changes are small. The proposal stands or falls on the latency–quality curve. If warm-starting with $k$ iterations matches our quality at our latency, the gated network adds nothing, and we will say so.
We expect it to fall short in two places. First, at sub-second budgets it has too few iterations to absorb large deformations. Second, during bursts of smoke, blood, or tool occlusion, gradient descent fits the occluder into the persistent scene and corrupts memory. A learned gate can instead learn to hold the old state. The occlusion-burst experiment below tests this directly.
Why not just re-fit EndoGaussian per clip, since two minutes is already fast? Two minutes per update, repeated across a multi-hour operation, means the displayed model is always minutes stale. Each re-fit also discards everything outside the current window, which is exactly the peripheral context that keyhole tunnel vision removes (Overcoming Intraoperative Visualization Constraints in Minimally Invasive Surgery). The revisit metric measures this loss directly.
Why not just fine-tune a feed-forward model like DGS-LRM on surgical video and run it per window? It removes the latency, but each window is reconstructed independently. Regions that leave the view vanish, and consecutive outputs flicker. DGS-LRM already reports degradation away from its input views on natural video, and monocular endoscope trajectories are narrow. Our memory-free ablation is exactly this alternative. If it matches our model on revisit frames, then the history is not doing work.
Why not just carry the old reconstruction forward? CRONOS warns that this can be surprisingly strong. It will be, wherever tissue is static. We report it on every metric, split by whether the region actually changed, so any gain we claim comes from the regions where carrying forward is wrong.
Proposed Experiments
We use ENDONERF in-vivo prostatectomy video, SCARED porcine laparoscopy, and the Hamlyn phantom and in-vivo sequences, with history/window splits at several gaps.
Image quality is measured with PSNR, SSIM, and LPIPS. SSIM is a structural similarity score; LPIPS is a learned perceptual distance. We report these separately for current-window held-out frames and for revisit frames. We also report update latency and the memory used. Drift is measured as revisit-frame PSNR after 10, 50, and 100 chained updates.
Freshness calibration is tested by checking whether low-freshness regions are exactly those whose appearance differs most between an EndoGaussian fit on early frames and one on late frames.
Three checks are decisive.
The first is the latency–quality curve against warm-started EndoGaussian. Our point must lie above the curve at sub-second latency.
The second is an occlusion-burst test. Synthetic smoke and tool masks are composited into a window, followed by a clean window. We measure how much each method's persistent scene is damaged on the clean window afterward.
The third is cross-dataset transfer: train on ENDONERF and SCARED, test on Hamlyn. This shows whether a single trained update network generalizes across surgical domains, which is the generalization failure the file attributes to feed-forward models trained outside surgery.
If the gate gives no gain over "keep everything and only add," or warm-starting wins at equal latency, the conclusion is that persistent memory matters but learned gating does not. That is still a useful, publishable negative result for how intraoperative reconstruction should be built.
References
- EndoGaussian: Gaussian Splatting for Deformable Surgical Scene Reconstruction
- Efficient Deformable Tissue Reconstruction via Orthogonal Neural Plane
- DGS-LRM: Real-Time Deformable 3D Gaussian Reconstruction From Monocular Videos
- DepthSplat: Connecting Gaussian Splatting and Depth
- GS-LRM: Large Reconstruction Model for 3D Gaussian Splatting
- DeltaNet: Conditional Medical Report Generation for COVID-19 Diagnosis
- Learning to Exploit Temporal Structure for Biomedical Vision-Language Processing
- CRONOS: Continuous time reconstruction for 4D medical longitudinal series
- Landmark-Free Preoperative-to-Intraoperative Registration in Laparoscopic Liver Resection
- Preoperative-to-intraoperative Liver Registration for Laparoscopic Surgery via Latent-Grounded Correspondence Constraints
- vNOTES hysterectomy: strategic repositioning of the vaginal route through technological evolution and emerging surgical variants