Amortized Preoperative-to-Intraoperative Deformation for Surgical Clearance Estimation

During keyhole surgery, the structures that most constrain what the surgeon can safely do cannot be seen by the camera. These include the pubic bone and coccyx that bound the pelvis in prostatectomy, and the sphincter and neurovascular bundles that lie just beneath the visible tissue. Preoperative CT and MRI do contain these structures, and measurements of them predict whether cancer will be left at the cut edge. The catch is that those scans are taken before the patient is positioned, before the abdomen is inflated with gas, and before instruments start pushing tissue around. Current methods that warp the preoperative anatomy to match the live endoscope work by running an optimizer over each scene. That takes minutes, and the result goes stale every time tissue moves.

We propose a single trained network that, in under a second, takes a handful of endoscopic frames and the patient's own preoperative surface meshes and predicts two things: how the soft organ has deformed relative to its preoperative shape, and where the camera sits relative to the rigid bone. From these we compute a live map of clearance, meaning the distance from visible tissue to hidden rigid boundaries. The idea borrows from longitudinal medical imaging. It treats the operating room as a "follow-up visit" to the preoperative "baseline" scan. The network predicts only the change from baseline, and it has to beat the strong do-nothing answer of "nothing changed."

Core Claim

Two features of the surgical setting make amortized prediction feasible where generic feed-forward reconstruction has struggled. Here "amortized" means a network trained once and run in a single pass, as opposed to optimization run per scene. The first feature is that bone does not deform. It gives a subsurface anchor that is known exactly in the preoperative frame, so the only rigid unknown is the camera-to-CT transform. The second feature is that soft tissue deforms within a limited family relative to a known patient-specific shape. The network therefore never has to invent geometry outside the camera's view, which is the regime where feed-forward models blur and hallucinate. It only has to predict a displacement field on a mesh it is given.

We claim this baseline-conditioned, change-only prediction gets within a small margin of optimization-based coupling of CT to Gaussian splatting on clearance error, at a fraction of the latency. Gaussian splatting represents a scene as many small colored 3D blobs that can be rendered quickly and differentiably. We also claim the network clearly beats rigid registration of the preoperative scan. We judge all methods on clearance error, not on how realistic the rendered image looks.

Relation to Prior Work

BridgeSplat supplies the representation we reuse. It attaches Gaussians to a CT-derived surface mesh through barycentric coordinates, which fixes each blob to a position within a mesh triangle. It then jointly optimizes mesh deformation and appearance against the endoscopic images, using an as-rigid-as-possible (ARAP) penalty that discourages unnatural stretching and shearing. On simulated tool pushing, pulling and dragging it reports an average Euclidean error within 5 mm. It also motivates our work through what it leaves open. It is explicitly not real-time. It assumes the initial rigid alignment is already given. It tracks only visible surfaces. We keep the rigged-mesh output but replace the optimizer with a trained predictor, and we make the rigid alignment an output of the network rather than an input.

A second group shows the cost of per-scene fitting in endoscopy. EndoGS, Endo-4DGS and SurgicalGS fit deformable Gaussian scenes per video. SurgicalGS reaches a depth RMSE of 2.174 mm on StereoMIS, and Endo-4DGS reports 4 to 7 minutes of optimization. None of them uses preoperative imaging, so none can report distance to anything the camera cannot see. EndoLRMGS is the closest feed-forward attempt. It uses a pretrained large reconstruction model only for rigid instruments and still optimizes the tissue per scene. We do the reverse: tissue is predicted feed-forward, conditioned on patient anatomy.

The feed-forward architecture comes from GS-LRM. It encodes posed images with per-pixel camera-ray coordinates in a transformer and predicts Gaussians in 0.23 s. We also borrow from DepthSplat, which showed that differentiable splatting can supervise geometry on unlabeled video. That gives us a route to self-supervised adaptation on real surgical footage.

The longitudinal literature contributes a lens and an evaluation discipline rather than an architecture. CRONOS showed that slowly changing anatomy makes "copy the last scan" a hard baseline to beat: 8.38 NRMSE for the last-context baseline versus 7.55 for CRONOS on the Lumiere glioma series. It also concedes that image fidelity is not surgical utility. DeltaNet encodes baseline and follow-up differences explicitly to suppress static anatomy. We adopt both ideas. The network predicts change relative to a patient baseline, and the rigidly registered preoperative mesh is the "last context" reference it must beat.

Finally, Fan et al. (PSM nomogram AUC 0.777) and the Chongqing RARP study (random-forest test AUC 0.97, five of seven selected features spatial) motivate why clearance to rigid pelvic structures matters clinically. Both measure workspace only before surgery, and they do it by hand.

Research Program

The model receives three inputs. The first is two to four endoscopic frames with approximate poses from robot kinematics. The second is the preoperative surface mesh of the deformable organ. The third is the segmented rigid bone surface. Image patches become tokens carrying camera-ray coordinates, as in GS-LRM. Mesh vertices become tokens carrying their preoperative position and normal. A transformer alternates attention within each set and across the two sets.

The network produces three outputs. It predicts a displacement for every organ vertex. It predicts a small rigid correction to the camera-to-CT transform; bone is fixed in the CT frame, so this correction alone places the hidden boundary in camera coordinates. It also predicts appearance parameters for Gaussians rigged to the deformed mesh in the BridgeSplat manner. Clearance is then read off directly, as the distance from each visible deformed-tissue point to the nearest bone point.

Training has two stages. The first is supervised pretraining on simulated interactions, where true displacements are known. We build these by combining CT-derived meshes with the DejaVu-style tool–tissue simulation used by BridgeSplat. We randomize insufflation, pushing, pulling and retraction, and render endoscopic views with specular and fluid appearance augmentation. The second stage is self-supervised adaptation on real video. On StereoMIS, EndoNeRF and SCARED sequences, which lack CT, we use photometric and stereo-depth losses through the differentiable renderer, plus the ARAP penalty. On these sequences only the appearance and deformation branches are active, and a pseudo-template mesh is taken from the first frame.

We plan an optional test-time refinement: a few optimizer steps started from the network's prediction. This lets us draw a single curve of latency against accuracy that connects our method to BridgeSplat.

The honest weak point is data. Paired CT, endoscopy and ground-truth deformation exist publicly only in simulation, in BridgeSplat's two porcine cases (where only qualitative results are reported), and in the static but exactly CT-aligned SERV-CT stereo pairs. The program is designed so that each claim is tested where ground truth actually exists.

Baselines

The do-nothing reference is rigid registration of the preoperative anatomy. The preoperative mesh is placed using robot kinematics, refined by rigid surface-to-depth alignment on stereo depth, and never deformed. This plays the role that the last-context image plays in CRONOS. Because bone is truly rigid and many organ regions move little, it will be strong on average and weak near instruments. We will run it on exactly the same frames and pose inputs as our model. Beating it means lower clearance error overall, and specifically in regions within 1 cm of instrument contact, with the difference tested across held-out simulated anatomies.

The strongest existing method is BridgeSplat's joint optimization, using its published formulation and ARAP weight. It receives the same frames, meshes and initial pose. It is given both its full iteration budget and matched wall-clock budgets of 0.5 s, 5 s and 60 s on the same GPU. The only reported figure we can anchor to is its average Euclidean error within 5 mm on simulated pushing, pulling and dragging. It reports no number on clinical pig data, and we found no reported clearance-error result on Lumiere or anywhere else in the file. We count it as beaten if our single pass is within 1 mm of its fully converged error while being at least one order of magnitude faster. We count it as clearly beaten if it cannot match our error under any budget below 5 s. Where applicable we also report SurgicalGS depth RMSE (2.174 mm on StereoMIS) as the per-scene reference for visible-surface accuracy. For cross-checking depth, the SCARED challenge results are around 1.71 mm and 0.42 mm mean absolute depth error on its two test sets.

The ablations isolate the proposal's key ingredients. The first removes preoperative conditioning: the same transformer predicts free depth-based Gaussians and the preoperative mesh is fitted to them afterward. The second predicts absolute vertex positions instead of change from baseline. The third freezes the camera-to-CT correction at the kinematic estimate, which isolates how much the learned rigid anchor matters. All variants share training data, parameter count and tuning budget.

Why not just…?

Why not just rigidly register the preoperative scan and skip deformation? For the bone this is nearly correct, which is why rigid registration is our do-nothing baseline rather than a straw man. BridgeSplat found persistent misalignment of rigid overlays under insufflation and tool retraction in porcine surgery. The deformable-splatting papers likewise document large non-rigid tissue motion. Whether that misalignment matters for clearance, and not just for image overlay, is an open question. Our stratified comparison near instrument contact answers it directly. If rigid registration matches us within 1 mm there, the deformation branch is unnecessary, and we would report that.

Why not just warm-start per-scene optimization and accept a few seconds? This is the real threat. Endo-4DGS takes 4 to 7 minutes from scratch, but a tracker started from the previous frame's solution might need only a few iterations. That is why BridgeSplat is run under matched wall-clock budgets, including a warm-started variant. The feed-forward model has two arguments in its favor that do not depend on raw speed. It estimates the rigid camera-to-CT correction that BridgeSplat assumes is given. It also does not accumulate drift after occlusion, which is the failure mode EndoLRMGS reports for optimization at tool–tissue boundaries. We will measure recovery after a simulated full occlusion. If warm-started optimization matches us at under 1 s with no drift, our contribution shrinks to initialization. We would say so, and the refinement curve shows exactly where that crossover falls.

Why not just run a general feed-forward reconstructor such as MapAnything or DepthSplat, fine-tuned on surgical video, and fit the preoperative mesh to its output? These models reconstruct only what lies in the camera's view and blur when asked to go beyond it, while clearance depends on geometry that is never visible. The no-conditioning ablation is exactly this pipeline with matched capacity. The hypothesis fails if it equals our model on clearance error.

Why not just use the preoperative workspace measurements, which already predict margins? Fan et al. and the Chongqing study predict a single patient-level risk before surgery. They say nothing about where, at a given moment, tissue is close to a rigid boundary. We see our output as complementary to those measurements rather than competing with them.

Proposed Experiments

The first decisive check uses held-out simulated anatomies, with shapes never seen in training, under pushing, pulling, dragging and insufflation. We report clearance error to bone, per-vertex displacement error and latency against all baselines. Error is stratified by distance from the instrument and by the fraction of the organ in view.

The second check uses SERV-CT. There the CT was acquired with the endoscope in place, so the true camera-to-CT transform and clearance are known exactly. We perturb the kinematic pose by realistic amounts and measure whether the learned rigid correction recovers it better than rigid surface alignment. The resulting error is compared against SERV-CT's roughly 2 mm reference depth accuracy, which is the floor below which differences are not meaningful.

The third check covers real deforming video without CT: StereoMIS, EndoNeRF and SCARED. We report visible-surface depth error against the per-scene methods, to confirm that amortization does not badly degrade what the camera can see. SurgicalGS's 2.174 mm on StereoMIS is the reference, and we accept a loss of at most about 1 mm in exchange for sub-second inference.

Finally, we evaluate qualitatively on BridgeSplat's two porcine cases with preoperative CT. No number is reported for those cases, so they serve only as a sanity check.

The proposal succeeds if two conditions hold together. It must beat rigid registration on contact-region clearance error in simulation. It must also stay within 1 mm of converged BridgeSplat at under a second, with no budget-matched optimizer doing better. It fails cleanly if either warm-started optimization or the unconditioned feed-forward ablation matches it.