TL;DR
SurgiSim automatically converts ordinary monocular surgical videos into interactive, physics-grounded 3D simulation environments. By combining geometrically regularized dynamic 3D Gaussian Splatting with a viscoelastic Material Point Method (MPM) and video-guided parameter estimation, it enables realistic, stable soft-tissue manipulation without manual 3D modeling.
What Problem Does This Paper Address?
Surgical training simulators and robotic surgical planners typically rely on commercial engines built on hand-crafted polygon meshes and manually tuned physics. Building these environments is labor-intensive and limits the variety of available training scenarios.
While recent neural scene representations—such as Neural Radiance Fields (NeRF) and 3D Gaussian Splatting (3DGS)—can reconstruct dynamic surgical scenes from monocular video, they are built solely for novel-view synthesis. Their deformation fields focus on matching visual appearance frame-by-frame rather than maintaining underlying physical consistency. When plugged into physics engines, these inconsistent geometries tear, fragment, or explode.
Furthermore, existing physics-integrated Gaussian methods (e.g., PhysGaussian) usually assume simple elastic or elastoplastic materials with uniform, hand-picked parameters. Real biological tissues, however, are viscoelastic: when perturbed, they dissipate energy and quickly return to rest rather than bouncing around like jelly.
Methods
SurgiSim bridges video reconstruction and interactive continuum mechanics through a three-part pipeline:
- Geometrically Consistent Canonical Reconstruction:
- Data Preprocessing: Segments instruments using SAM, inpaints obscured tissue regions, and predicts monocular depth with Depth Anything v2.
- Trajectory & Geometric Regularization: A 3DGS canonical model is animated through a multi-resolution 4D deformation field. To prevent Gaussian kernels from crossing paths or interleaving, a multi-stage optimization applies a trajectory loss ($\mathcal{L}_{\text{traj}}$) that enforces local motion to be parallel and smooth. An anisotropic penalty ($\mathcal{L}_{\text{geo}}$) limits Gaussian aspect ratios ($\le 3$) and scales to eliminate spiky surface artifacts.
- Surface Thickening: Because monocular reconstruction only yields a hollow 2.5D surface, an algorithm samples points along the depth axis into an internal bounding volume to provide the particle volume required by volumetric physics.
- Viscoelastic Deformation via MPM:
- Uses the hybrid Lagrangian-Eulerian Material Point Method (MPM).
- Instead of purely elastic models, SurgiSim couples a fixed corotated elastic energy with a Maxwell-based viscous dissipation potential ($\Psi(\partial F_v/\partial t) = \frac{1}{2}\eta_v \operatorname{tr}((\partial F_v/\partial t)^\top \partial F_v/\partial t)$) to realistically damp rapid rebounds and simulate energy loss.
- Physics-Guided Parameter Estimation:
- Tracks 2D tissue contact points near surgical tools via optical tracking (DOT) and lifts them to 3D via predicted depth.
- Drives a differentiable MPM simulation and rasterizer with this trajectory, minimizing photometric discrepancy ($\mathcal{L}_v$) between the rendered simulation and real video frames over a rolling time window to optimize material parameters ($\mu_E, \eta_v, \gamma_v$).
[Monocular Video]
│
├─► SAM + Inpainting + Depth Estimation
├─► Multi-Stage 3DGS (Trajectory & Geometric Regularization) + Surface Thickening
│ └─► Geometry-Consistent Canonical 3DGS Simulation Scene
│
└─► 3D Contact Motion Tracking
└─► Differentiable MPM & Rasterization
└─► Inferred Viscoelastic Parameters (Shear & Viscosity)
Datasets
Evaluations were conducted on the public EndoNeRF dataset (Wang et al., 2022), consisting of stereo endoscopic video sequences of robotic surgery displaying non-rigid tissue deformations. The authors used 5 sequences (spanning 4–8 seconds, ~807 frames total) to establish 5 simulation environments across 10 typical surgical interactions. (One sequence featuring extensive tissue cutting was excluded). No new benchmarks were introduced.
Key Insights and Ideas
- Visual accuracy does not equal physical plausibility: A deformation field can easily overfit to synthesize sharp frames while allowing Gaussian splats to swap positions or stretch infinitely. Enforcing trajectory parallelism ensures that space does not fold onto itself.
- Tissues need viscous damping: Purely elastic simulations fail on organs because tissue fluids cause significant viscous drag. Introducing a Maxwell-type dissipation tensor eliminates artificial wobbling.
- Grounding parameters in observed video motion beats generative priors: Rather than using diffusion-based priors (like Score Distillation Sampling), which often hallucinate or fail on specialized surgical tasks, inverting observed tool-tissue interactions directly via differentiable MPM yields accurate physical parameters.
Key Results
- User Study (68 participants):
- Evaluated across 44 professional surgeons and 24 non-medical laypersons.
- SurgiSim was preferred 57.8% of the time by surgeons and 77.8% by laypersons, dramatically outperforming SimEndoGS (25.6% / 21.3%), point-based MPM (10.0% / 0.5%), and mesh-based MPM (6.7% / 0.4%).
- Simulating with video-guided parameter estimation was preferred over unguided defaults by 65.6% of surgeons and 82.9% of laypersons.
- Visual Fidelity & Dynamics:
- Reconstructs initial canonical frames with 37.11 dB PSNR, 0.977 SSIM, and 0.044 LPIPS.
- Y-T spatiotemporal slice analyses confirmed that SurgiSim-modeled tissue settles into a quiescent state immediately after tool release, whereas baseline elastic formulations exhibit prolonged, unnatural oscillations.
Limitations and Future Work
- Hidden Geometry & Flaws: Because only visible surfaces are recorded by monocular endoscopy, the heuristic surface-thickening method can produce inaccurate textures or properties on invisible back sides.
- Motion Sparsity: If a video segment only features movement in a small localized area, there is insufficient dynamic evidence to infer material properties across the rest of the organ.
- Topological Changes: The current framework cannot model irreversible cutting, tearing, or topological dissection.
Summary updated