Physics-Surrogate Deformation Spaces for CT-Rigged Gaussian Splatting

During minimally invasive surgery, organs move and bend. Breathing, gas inflation of the abdomen, and instruments pushing on tissue shift anatomy by centimeters away from the preoperative CT scan. Surgeons need the positions of structures they cannot see, such as a tumor or a vessel inside the liver, but the endoscope only shows the organ's wet, partly occluded outer surface. Recent methods can track that visible surface well by representing the scene as Gaussian splats, which are clouds of small colored ellipsoids that can be rendered and optimized quickly. They have no principled way to infer what happens inside the organ.

We propose to fix the space of allowed deformations before looking at any video. From a patient's segmented CT, we train a fast neural stand-in for a finite-element simulation (FEM, the standard numerical solver for elastic tissue). This network maps a small set of load parameters to a full volumetric displacement field, and it assigns different stiffness to parenchyma, tumor, and vessels. Intraoperatively, Gaussians are attached to this volumetric field, not to a surface mesh, and the photometric rendering loss optimizes only the few load parameters. Every surface motion the video supports therefore comes with an interior motion that obeys the patient's heterogeneous mechanics. The same displacement field then warps the Gaussians, the CT voxels, and the tumor mesh in one step. This is the practical form of "representation heterogeneity" in this setting: one continuous field serves every format the clinic uses.

Core Claim

Coupling the video to a volumetric, material-heterogeneous deformation space during optimization should reduce the error in locating interior targets. Adding physics after the surface has been tracked should do worse, and the gap should be largest when much of the organ surface is occluded or unobserved and when stiff inclusions are present. We also expect the coupling to reduce the number of free parameters from thousands of vertex offsets to tens of load coefficients. That should make per-frame optimization cheaper and less prone to physically impossible solutions such as tearing or volume loss. If a two-stage pipeline matches us on interior error, the claim is wrong. The experiments are built so that this comparison is the central one.

Relation to Prior Work

The closest work is BridgeSplat. It attaches Gaussians to a CT-derived surface mesh using barycentric coordinates, meaning each Gaussian's position is expressed relative to a mesh triangle. It then jointly optimizes appearance and vertex displacements under an as-rigid-as-possible penalty and pushes the surface displacement back into the CT volume. Its authors note that it mainly captures surface deformation and that deeper dynamics would need biomechanical modeling. We keep its idea of rigging Gaussians to preoperative anatomy and change two things. The rig becomes volumetric, and the free deformation becomes a learned physical surrogate instead of vertex offsets with a uniform rigidity penalty.

V2S-Net supplies the method we reuse. It shows that a 3D convolutional network trained on 80,000 FEM-deformed synthetic shapes can map a preoperative distance field plus a partial intraoperative surface to a dense displacement field in 130 ms. It does this with homogeneous random materials, and it needs a separate stereo surface reconstruction step with no photometric feedback. We adopt its synthetic-FEM training strategy, add per-label stiffness, and replace its reconstructed-surface input with gradients from differentiable rendering.

Surface-only video methods motivate the proposal by showing the gap. Endo-4DGS fits monocular endoscopic video with Gaussians moved by small networks and depth priors, and reaches real-time rendering. Land-Reg registers a liver model to single laparoscopic frames using matched landmarks and a surface isometry penalty. Neither models the interior. The P2ILF challenge analysis reports that existing registration pipelines remain largely rigid and fail from poor initialization. That result constrains us: we do not claim to solve cold-start initialization, and we assume a rough rigid alignment exactly as BridgeSplat does.

VoMP predicts volumetric stiffness and density fields from meshes, radiance fields, or splats by lifting 2D image features into a voxel grid. The obvious bridge at this intersection would be to run a VoMP-style predictor on the reconstructed organ and use its output to drive the deformation. We deliberately reject that design, for reasons given below.

Research Program

The pipeline has two parts. Everything expensive happens preoperatively.

The preoperative part starts from the patient's segmented CT label volume $A$, which marks parenchyma, tumor, and vessels. We assign stiffness ranges per label, drawn from literature ranges rather than from image appearance, and build a tetrahedral mesh. We then run hyperelastic FEM under many random surface loads, including respiratory-like boundary conditions and localized tool pushes, pulls, and drags. From these runs we train a surrogate $G(A, f)$ that maps the anatomy and a low-dimensional load code $f$ to a displacement field $u(x)$ defined everywhere in the organ. The load code is compressed from the simulated force patterns with an autoencoder. The surrogate is conditioned on $A$ so that a single network, trained across many anatomies in the style of V2S-Net plus random embedded inclusions, can be fine-tuned per patient in minutes rather than retrained from scratch.

The intraoperative part initializes Gaussians on the visible organ surface and inside a thin shell beneath it. Each Gaussian is attached to a material point $x_i$ of the volume. A Gaussian with rest center $x_i$ and covariance $\Sigma_i$ is moved to $x_i + u(x_i)$ and gets covariance $J_i \Sigma_i J_i^\top$, where $J_i$ is the Jacobian of the deformation at $x_i$. Rendering losses therefore flow into $f$ through a physically structured map. Per frame we solve

$$ \min_{f_t,\, T_t,\, \theta} \; \mathcal{L}_{\text{photo}}(f_t, T_t, \theta) + \lambda_d \mathcal{L}_{\text{depth}} + \lambda_t \lVert f_t - f_{t-1} \rVert^2 + \lambda_r \mathcal{L}_{\text{res}}(r_t). $$

The terms are defined as follows. $f_t$ is the load code at frame $t$. $T_t$ is the camera pose. $\theta$ collects the Gaussian appearance parameters. $\mathcal{L}_{\text{photo}}$ is the rendering error. $\mathcal{L}_{\text{depth}}$ is a confidence-weighted monocular depth prior as in Endo-4DGS. The third term penalizes load codes that jump between frames.

The surrogate will not capture everything, for example cutting, fluid, or modeling error. We therefore allow a small residual displacement $r_t$ that is added only to surface Gaussians. $\mathcal{L}_{\text{res}}$ penalizes its size, so any motion the physics cannot explain stays visibly on the surface and is never silently propagated inward. The magnitude of $r_t$ doubles as a warning signal: large residuals mean the physics prior is being violated, and the interior estimate should be trusted less.

Once $f_t$ is found, the same field warps the CT voxels, the tumor mesh, and the vessel centerlines. This gives the interior overlay that navigation needs, with no separate propagation step.

Baselines

The first reference point is doing nothing: a rigid overlay using the initial alignment and the tracked camera pose. Any method that does not beat it on interior error is useless clinically.

The strongest existing competitor is BridgeSplat. It addresses the same input (preoperative CT plus monocular video) and already produces a deformed CT. On the DejaVu tool-tissue simulation it reports an average Euclidean distance error within 5 mm across pushing, pulling, and dragging. That figure is for tracked surface vertices, and no interior-target number is reported on Lacuna. We will rerun it from its public description on identical sequences, with identical initial rigid alignment, identical Gaussian counts, and the same per-frame optimization iteration budget and GPU. Its as-rigid-as-possible weight will be tuned on the same validation sequences we use to tune $\lambda_r$ and $\lambda_t$. Beating it means lower interior target error, measured on tumor centroid and vessel bifurcations, with surface error no worse.

The most dangerous baseline is a two-stage pipeline built from existing parts. BridgeSplat tracks the surface. Its tracked visible surface is then given to V2S-Net, retrained on our same heterogeneous FEM data for fairness, to infer the interior. On real human breathing data, V2S-Net reports internal vessel-bifurcation errors of 4.8 to 5.7 mm with 130 ms inference.

Three ablations isolate our ingredients. The first uses our full model with homogeneous stiffness, which tests whether material heterogeneity matters. The second attaches Gaussians to a volumetric tetrahedral mesh with a volumetric as-rigid-as-possible penalty and no learned surrogate, which tests whether learned physics beats a geometric prior. The third removes the residual term, which tests whether the escape valve protects interior estimates.

On the single-image P2ILF setting, the reference is Land-Reg. It reports a non-rigid Dice of 75.38% and a mean target registration error of 76.46 pixels.

Why not just…?

Why not just predict tissue stiffness with a VoMP-style model from the reconstructed organ? VoMP learns stiffness from appearance features lifted from rendered views. It was trained on furniture and household objects, and it assumes isotropic materials. An embedded tumor is by definition not visible from the organ surface, so appearance cannot reveal the stiffness contrast that matters most. CT segmentation already tells us where the tumor is. For this reason we assign stiffness ranges per segmented label and randomize within those ranges during simulation, rather than predicting stiffness from images. We will report sensitivity to deliberately wrong stiffness assignments so this choice is tested rather than assumed.

Why not just track the surface with BridgeSplat and then complete the interior with a volumetric solver? This is the real threat, and it is the central comparison. Our argument is that when the surface is heavily occluded by tools or blood, or lies outside the field of view, surface tracking under a uniform rigidity penalty hallucinates motion in the unseen regions. A downstream interior solver then inherits those errors. A physics-constrained search space should instead fill unseen surface regions consistently with the loads that explain the seen ones. To test this, we will stratify results by the fraction of the organ surface visible and by the magnitude of tool contact. If the two-stage pipeline matches us across all strata, we will conclude that coupling is unnecessary and report that.

Why not just use a volumetric as-rigid-as-possible penalty on a tetrahedral mesh? It is cheap and needs no training, and on near-homogeneous tissue under small deformation it may be just as good. The ablation described above answers this. We expect the surrogate to help mainly around stiff inclusions and under large tool-induced loads, where uniform local rigidity is simply the wrong physics. If that expected advantage does not appear, the learned surrogate is not worth its preoperative cost.

Proposed Experiments

The decisive experiment needs interior ground truth, and no dataset in our file provides it at scale. We will therefore build simulated sequences from segmented abdominal CT anatomies. Each will contain embedded stiff inclusions, respiratory and tool loads, and ground truth generated with a finer FEM mesh and a different constitutive model than the surrogate was trained on, so that we do not evaluate the surrogate against itself. The sequences will be rendered photorealistically with specularity, smoke, and tool occlusion, and we will add DejaVu-style push, pull, and drag interactions. Metrics are interior target registration error in millimeters, surface error, volume change, the per-frame time to converge, and the correlation between residual magnitude and interior error. The key check is the interaction between visible-surface fraction and method: our advantage over the two-stage pipeline should widen as visibility drops. The homogeneous-stiffness ablation should lose specifically on inclusion-centroid error.

Real data will test transfer. On the visceral pig dataset from BridgeSplat, only qualitative results are reported, so we will add held-out-frame photometric error and tracking of annotated surface landmarks. We will pursue interior validation there only if follow-up imaging, such as intraoperative cone-beam CT, can be obtained; otherwise we will state that it is missing. On EndoNeRF and StereoMIS we will check that constraining motion to the physical space does not harm rendering quality relative to Endo-4DGS. Endo-4DGS reports no PSNR values on Lacuna for these sets, so we will rerun it. On P2ILF we will run a single-frame version against Land-Reg's reported 75.38% Dice and 76.46-pixel error, understanding that 2D metrics only partly reflect interior accuracy.

The project fails in an informative way under either of two outcomes: the surrogate's modeling error dominates real tissue behavior, visible as large residuals on pig data, or the two-stage baseline matches us under occlusion. Either result would tell the field whether coupling physics to video during optimization matters, or whether adding it afterwards is enough.