Two-Speed Gaussian Maps for Streaming Reconstruction of Deforming Surgical Tissue

Surgeons need a live 3D model of the tissue in front of the endoscope. With it they can overlay preoperative scans, guide robots, and measure anatomy. Current methods split into two camps, and neither delivers this. Methods that model deforming tissue with Gaussian splatting represent the scene as many small colored 3D blobs that are moved over time and rendered quickly. They reach high image quality, but only after optimizing over a whole recorded clip. Streaming methods, which estimate camera motion and build the map as frames arrive, either assume the tissue is rigid or accumulate large errors once it moves. Their position error grows from 0.23 mm on a static phantom to 15.47 mm on breathing pig tissue in EndoFlow-SLAM. Visual object tracking has faced the same problem for years under the name stability-plasticity dilemma. A model that updates on every frame absorbs occluders and glare into what it believes is the target, and then drifts further with each update ("model drift"). A model that updates too little cannot follow real change.

We propose to give every Gaussian two copies of its geometry. A fast state follows the current frame. A slow state stores the tissue's rest shape. The fast state is pulled back toward the slow state, with a strength set by how much the current observation can be trusted. The slow state changes only when a deformation persists and does not reverse, as happens after a cut or a retraction that stays in place. Breathing and brief tool pushes then do not rewrite the map, glare and blood do not corrupt it, and permanent surgical changes still enter it. The slow state also serves as a stable reference for estimating camera motion.

Core Claim

We claim that, in streaming surgical reconstruction, the key decision is per-Gaussian and three-way: is a disagreement between the map and the image transient corruption, reversible deformation, or persistent change? Making this decision explicitly reduces long-horizon drift in both the camera trajectory and tissue positions, compared with any single fixed regularization strength.

A fixed pull toward a rest shape cannot do this. A strong pull "melts" cuts back toward the old shape. A weak pull lets breathing and tool contact leak into the map until it drifts. Freezing the map during bad frames cannot follow tissue that deforms while it is occluded. The claim is falsified if a well-tuned fixed-weight rest-shape penalty matches the two-speed model on the tests below. We treat that comparison as the decisive experiment.

Relation to Prior Work

Three groups of prior work matter here. Each plays a different role.

The closest existing system, and the strongest motivation, is EndoMD-SLAM. It splits the rendered image into a global static field and a per-view transient field that absorbs lens contamination. A reliability gate pauses map updates during events like lens flushing. On degraded C3VDv2 sequences this cuts trajectory error from about 35 mm to 3.02 mm. The authors state that it handles only rigid or semi-rigid scenes and name deformable splatting as the next step. Its gate makes a binary freeze/update choice. Ours is continuous, per Gaussian, and separates two kinds of "update": following a reversible deformation versus consolidating a change into the rest shape. EndoGSLAM is a fast rigid streaming splatting system running at 15–16 frames per second. It contributes the efficiency tricks we reuse: simple per-Gaussian color and optimization limited to local keyframes. Its rigidity assumption is the thing we remove. EndoFlow-SLAM adds per-Gaussian optical-flow losses to a streaming system. It is the only streaming deformable baseline in the file with numbers on breathing tissue. Its drift on StereoMIS is the gap we target.

The second group consists of offline deformable reconstructors: EndoGS, Deform3DGS, HFGS, Endo-4DGS and the plane-based Forplane. They achieve high quality, up to 37.90 dB PSNR for Deform3DGS and 38.14 dB for HFGS on EndoNeRF. However, they fit deformation over a fixed, pre-recorded time span, using either temporal feature planes or learnable basis functions in time. None of them is causal. We use them as the quality ceiling a streaming method should approach. Endo-4DGS also supplies the monocular depth-prior recipe we use when stereo is unavailable.

The third group supplies the method. Re2EMA recasts the moving-average template update as an optimization with a data term and a penalty pulling the model back toward the trusted first frame, controlled by a soft confidence function. It improves BACF AUC on OTB-2015 from 63.2% to 65.2%. Clustering of static-adaptive correspondences keeps keypoints anchored to the first frame alongside keypoints matched frame to frame. Adaptive correlation filters with long- and short-term memory runs a fast filter and a conservative identity filter in parallel. Deformable tracking with gated fusion shows that unconstrained deformable features "deform onto" occluders and shadows unless a learned gate falls back to rigid features. That is a negative result which constrains our design: plasticity must be gated.

All four of these tracking methods work on 2D boxes or sparse keypoints. All anchor to a fixed first frame. Surgery breaks a fixed anchor, because a cut permanently changes the target. Our consolidation rule is what lets the anchor itself move, and it is the part with no counterpart in these papers. LiteTracker shows that causal tissue tracking at 29.67 ms per frame is possible using a feature buffer and moving-average velocity initialization. We borrow its velocity prior to initialize fast states. It remains a 2D point tracker with no map and no anchor.

Research Program

Each Gaussian $i$ has a slow rest position $c_i$ and a fast offset $d_i(t)$, so its position at frame $t$ is $\mu_i(t) = c_i + d_i(t)$. Rotation and scale follow the same split. When a new frame arrives, we jointly update the camera pose and the fast offsets of Gaussians visible near the current keyframe. The objective is

$$ \min_{\{d_i\},\,T_t}\; \mathcal{L}_{\text{img}}(T_t, \{c_i + d_i\}) \;+\; \sum_i \lambda\,\big(1 - r_i(t)\big)\,\|d_i\|^2 \;+\; \beta \sum_{(i,j)\in\mathcal{N}} \|d_i - d_j\|^2 . $$

Here $T_t$ is the camera pose and $\mathcal{L}_{\text{img}}$ is the usual photometric, depth and per-Gaussian flow loss from EndoFlow-SLAM. The reliability $r_i(t)\in[0,1]$ measures how far the observation of Gaussian $i$ can be trusted. $\mathcal{N}$ is a k-nearest-neighbor graph over rest positions, and the term over it is a local-coherence penalty in the spirit of the "chain of trust" in the static-adaptive keypoint work. When reliability is low, the offset is pulled toward zero, meaning the Gaussian falls back to its rest shape, just as Re2EMA pulls a corrupted template toward its anchor. When reliability is high, the image term dominates and the Gaussian follows the tissue. Camera pose is estimated mostly from the rest geometry, because unreliable or freshly deformed Gaussians carry little weight in $\mathcal{L}_{\text{img}}$ after reweighting by $r_i$.

Reliability is computed without learning in the first version. It combines four signals: a surgical-tool mask, saturation detection for specular highlights (as in EndoGSLAM's lighting-aware filtering), forward-backward flow consistency, and the size of the Gaussian's rendering residual relative to its own running history. A learned gate that predicts $r_i$ from local features, in the style of gated fusion, is a second variant. It is compared against the heuristic version, not assumed to be better.

Consolidation runs every $W$ frames. For each Gaussian we compute a persistence statistic $\rho_i$: the norm of the mean offset over the window divided by the offset's standard deviation plus a small constant. It is near zero for oscillating breathing motion or brief pushes that return. It is large for an offset that has stayed displaced. The rest state is then updated as

$$ c_i \leftarrow c_i + \eta\,\bar r_i\,\sigma(\rho_i - \tau)\,\bar d_i, \qquad d_i \leftarrow d_i - \eta\,\bar r_i\,\sigma(\rho_i - \tau)\,\bar d_i , $$

where $\bar d_i$ and $\bar r_i$ are the window means of the offset and reliability, $\sigma$ is a sigmoid, $\tau$ is a persistence threshold, and $\eta$ is a consolidation rate. The update only re-splits the position between the two states, so the rendered image is unchanged at the moment of consolidation. Gaussians that stay unreliable and unseen for long periods keep their old rest state rather than being deleted. This addresses the problem in offline methods where tools hide tissue and the hidden region is corrupted. New Gaussians are added in unmapped regions, as in EndoGSLAM, and are seeded at their first reliable observation.

The engineering target is causal operation at 15 frames per second or faster, comparable to EndoGSLAM-R, on a single RTX 3090-class GPU. We do not claim 30 frames per second in advance.

Baselines

The naive reference is the same streaming deformable system with only a fast state and no pull toward a rest shape, that is, $\lambda=0$ and no consolidation. This is essentially flow-constrained streaming splatting in the style of EndoFlow-SLAM. We also run EndoFlow-SLAM itself as published. Its reported StereoMIS numbers are ATE 15.47 mm, PSNR 21.96 dB and SSIM 0.59 (EndoFlow-SLAM). It is the strongest existing streaming deformable competitor, so beating it on StereoMIS trajectory error by a clear margin is the main bar. Our target is a halving of ATE. A reduction within run-to-run variance would count as a failure. All streaming systems receive the same frames, the same depth input (stereo where available, Endo-4DGS-style monocular priors otherwise), the same GPU, and the same per-frame time budget. Pose-tracking hyperparameters are tuned on one held-out sequence per dataset with equal search budgets for every method.

The strongest existing anti-drift system is EndoMD-SLAM. It reports ATE 3.02 mm on degradation-filtered C3VDv2 at 0.95 FPS (EndoMD-SLAM). It is the right competitor on corrupted frames, but it assumes rigidity. We run it on the 15 C3VDv2 sequences deformed with a blood-pressure cuff and on StereoMIS. Beating it means lower ATE on deforming sequences while staying within 1 mm of it on the rigid degraded sequences.

For reconstruction quality, the offline ceiling is Deform3DGS at 37.90 dB PSNR on EndoNeRF (Deform3DGS), with HFGS at 38.14 dB (HFGS). We do not expect to beat these in a single causal pass. Success is coming within about 2 dB while running causally. We also measure Deform3DGS refit on sliding windows of recent frames, as a naive way to make an offline method "online."

Three ablations isolate the key ingredient. The first is a fixed-weight rest-shape penalty: constant $\lambda$, no reliability, no consolidation, with $\lambda$ swept over a grid with the same tuning budget as our method. The second is reliability without consolidation: the rest state is frozen at the first observation, which amounts to a direct 3D transplant of Re2EMA. The third is consolidation without reliability. The proposal works only if the full model beats the best point on the fixed-weight sweep on both the drift test and the cut test described below. Winning one while losing the other means the problem is just a single tradeoff that a tuned constant already handles.

Why not just…?

Why not just add a rest-shape regularizer with a tuned constant weight? This is the most serious threat. On sequences with only breathing, a well-tuned constant may match us, because breathing is reversible and any moderate pull helps. Our argument rests on sequences that mix reversible and permanent change. EndoNeRF contains cutting and tearing, and StereoMIS contains long breathing segments. We predict that the weight best for StereoMIS drift is worse on post-cut geometry in EndoNeRF, and the reverse. The sweep in the ablation either shows this crossing or does not. If it does not, we will report that the two-speed state is unnecessary and the contribution reduces to a tuned regularizer for streaming deformable mapping.

Why not just freeze mapping when frames look bad, as EndoMD-SLAM does? Freezing is safe for a rigid scene but discards information in a deforming one. Tissue keeps breathing while a tool or blood covers it, so on unfreeze the map is out of date and pose estimation must absorb the gap. EndoMD-SLAM's own limitation statement concedes the deformable case. Its reliability gate is also binary for the whole frame, while occlusion in surgery is usually partial. Running it on the C3VDv2 deformation sequences, and on our system with per-frame binary freezing in place of per-Gaussian $r_i$, settles this directly.

Why not just run an offline deformable method on a rolling window? Windowed refitting of Deform3DGS is cheap, since its full training takes about 64 seconds. However, each window re-solves deformation from scratch, has no memory of geometry hidden by tools in earlier windows, and must pick a reference pose for each window, which creates seams between windows. We expect rolling refits to show jumps in tissue position at window boundaries on STIR tattoo tracks, and loss of geometry under long occlusions. We measure both. If windows of a few seconds turn out seamless and fast enough, that is a strong simple competitor, and we would report it as such.

Why not just treat deforming tissue as outliers in a rigid SLAM system? In the abdomen almost every visible surface moves with breathing, as Feuerstein et al. documented early for insufflation-driven organ shifts (Automatic Patient Registration for Port Placement). Little static structure is left to track. Rigid systems also produce only sparse or rigid maps, which cannot support overlays on moving tissue. EndoGSLAM, a rigid baseline, is included on StereoMIS to quantify this.

Proposed Experiments

The first decisive check is long-horizon drift on StereoMIS. We report ATE over full sequences and its growth curve over time, for our model, the naive fast-only model, EndoFlow-SLAM, EndoGSLAM and the ablations. The prediction is that the growth curve flattens once the rest state forms, while the fast-only curve keeps rising.

The second check measures tissue-point drift on the STIR Challenge 2024 sequences. We read the positions of Gaussians nearest each infrared tattoo at the first frame and project them forward, then measure the error against tattoo ground truth at the end of the sequence. LiteTracker serves as a 2D point-tracking reference. For STIR, no reported result was found on Lacuna for an endpoint error figure. LiteTracker reports only its speedup and latency there, so we will run it ourselves. SuPer adds an occlusion-focused test, where LiteTracker reports 97.45% occlusion accuracy.

The third check is the cut test on EndoNeRF clips with cutting and tearing. We compare PSNR and depth error on frames after each cut, against Deform3DGS and HFGS as offline ceilings. We also add a targeted "melt" measure: depth error inside the incised region in the final frames, where a rest-shape pull that is too strong would visibly close the cut.

The fourth check is optical corruption combined with deformation, on the C3VDv2 blood-pressure-cuff deformation sequences and their paired clean and dirty passes. This tests whether per-Gaussian reliability keeps debris and dye out of the rest state, compared with EndoMD-SLAM's frame-level freeze.

Across all four, we log frames per second under the shared time budget. A result that wins on accuracy only by dropping below 10 frames per second will be reported as such, not as a real-time result.