Realistic Surgical Simulation from Monocular Videos

Kailing Wang, Chen Yang, Keyang Zhao, Xiaokang Yang, Wei ShenSubmission to ICLR 2025 · Withdrawn

TL;DR

SurgiSim automatically converts ordinary monocular surgical videos into interactive, physics-grounded 3D simulation environments. By combining geometrically regularized dynamic 3D Gaussian Splatting with a viscoelastic Material Point Method (MPM) and video-guided parameter estimation, it enables realistic, stable soft-tissue manipulation without manual 3D modeling.

What Problem Does This Paper Address?

Surgical training simulators and robotic surgical planners typically rely on commercial engines built on hand-crafted polygon meshes and manually tuned physics. Building these environments is labor-intensive and limits the variety of available training scenarios.

While recent neural scene representations—such as Neural Radiance Fields (NeRF) and 3D Gaussian Splatting (3DGS)—can reconstruct dynamic surgical scenes from monocular video, they are built solely for novel-view synthesis. Their deformation fields focus on matching visual appearance frame-by-frame rather than maintaining underlying physical consistency. When plugged into physics engines, these inconsistent geometries tear, fragment, or explode.

Furthermore, existing physics-integrated Gaussian methods (e.g., PhysGaussian) usually assume simple elastic or elastoplastic materials with uniform, hand-picked parameters. Real biological tissues, however, are viscoelastic: when perturbed, they dissipate energy and quickly return to rest rather than bouncing around like jelly.

Methods

SurgiSim bridges video reconstruction and interactive continuum mechanics through a three-part pipeline:

  1. Geometrically Consistent Canonical Reconstruction:
  2. Data Preprocessing: Segments instruments using SAM, inpaints obscured tissue regions, and predicts monocular depth with Depth Anything v2.
  3. Trajectory & Geometric Regularization: A 3DGS canonical model is animated through a multi-resolution 4D deformation field. To prevent Gaussian kernels from crossing paths or interleaving, a multi-stage optimization applies a trajectory loss ($\mathcal{L}_{\text{traj}}$) that enforces local motion to be parallel and smooth. An anisotropic penalty ($\mathcal{L}_{\text{geo}}$) limits Gaussian aspect ratios ($\le 3$) and scales to eliminate spiky surface artifacts.
  4. Surface Thickening: Because monocular reconstruction only yields a hollow 2.5D surface, an algorithm samples points along the depth axis into an internal bounding volume to provide the particle volume required by volumetric physics.
  5. Viscoelastic Deformation via MPM:
  6. Uses the hybrid Lagrangian-Eulerian Material Point Method (MPM).
  7. Instead of purely elastic models, SurgiSim couples a fixed corotated elastic energy with a Maxwell-based viscous dissipation potential ($\Psi(\partial F_v/\partial t) = \frac{1}{2}\eta_v \operatorname{tr}((\partial F_v/\partial t)^\top \partial F_v/\partial t)$) to realistically damp rapid rebounds and simulate energy loss.
  8. Physics-Guided Parameter Estimation:
  9. Tracks 2D tissue contact points near surgical tools via optical tracking (DOT) and lifts them to 3D via predicted depth.
  10. Drives a differentiable MPM simulation and rasterizer with this trajectory, minimizing photometric discrepancy ($\mathcal{L}_v$) between the rendered simulation and real video frames over a rolling time window to optimize material parameters ($\mu_E, \eta_v, \gamma_v$).
[Monocular Video] 
   │
   ├─► SAM + Inpainting + Depth Estimation
   ├─► Multi-Stage 3DGS (Trajectory & Geometric Regularization) + Surface Thickening
   │      └─► Geometry-Consistent Canonical 3DGS Simulation Scene
   │
   └─► 3D Contact Motion Tracking 
          └─► Differentiable MPM & Rasterization 
                 └─► Inferred Viscoelastic Parameters (Shear & Viscosity)

Datasets

Evaluations were conducted on the public EndoNeRF dataset (Wang et al., 2022), consisting of stereo endoscopic video sequences of robotic surgery displaying non-rigid tissue deformations. The authors used 5 sequences (spanning 4–8 seconds, ~807 frames total) to establish 5 simulation environments across 10 typical surgical interactions. (One sequence featuring extensive tissue cutting was excluded). No new benchmarks were introduced.

Key Insights and Ideas

  • Visual accuracy does not equal physical plausibility: A deformation field can easily overfit to synthesize sharp frames while allowing Gaussian splats to swap positions or stretch infinitely. Enforcing trajectory parallelism ensures that space does not fold onto itself.
  • Tissues need viscous damping: Purely elastic simulations fail on organs because tissue fluids cause significant viscous drag. Introducing a Maxwell-type dissipation tensor eliminates artificial wobbling.
  • Grounding parameters in observed video motion beats generative priors: Rather than using diffusion-based priors (like Score Distillation Sampling), which often hallucinate or fail on specialized surgical tasks, inverting observed tool-tissue interactions directly via differentiable MPM yields accurate physical parameters.

Key Results

  • User Study (68 participants):
  • Evaluated across 44 professional surgeons and 24 non-medical laypersons.
  • SurgiSim was preferred 57.8% of the time by surgeons and 77.8% by laypersons, dramatically outperforming SimEndoGS (25.6% / 21.3%), point-based MPM (10.0% / 0.5%), and mesh-based MPM (6.7% / 0.4%).
  • Simulating with video-guided parameter estimation was preferred over unguided defaults by 65.6% of surgeons and 82.9% of laypersons.
  • Visual Fidelity & Dynamics:
  • Reconstructs initial canonical frames with 37.11 dB PSNR, 0.977 SSIM, and 0.044 LPIPS.
  • Y-T spatiotemporal slice analyses confirmed that SurgiSim-modeled tissue settles into a quiescent state immediately after tool release, whereas baseline elastic formulations exhibit prolonged, unnatural oscillations.

Limitations and Future Work

  • Hidden Geometry & Flaws: Because only visible surfaces are recorded by monocular endoscopy, the heuristic surface-thickening method can produce inaccurate textures or properties on invisible back sides.
  • Motion Sparsity: If a video segment only features movement in a small localized area, there is insufficient dynamic evidence to infer material properties across the rest of the organ.
  • Topological Changes: The current framework cannot model irreversible cutting, tearing, or topological dissection.

Summary updated

Abstract

This paper tackles the challenge of automatically constructing realistic surgical simulation systems from readily available surgical videos. Recent efforts have successfully integrated physically grounded dynamics within 3D Gaussians to perform high-fidelity simulations in well-reconstructed static simulation environments. However, they struggle with the geometry inconsistency of simulation environments and unrealistic physical deformations of soft tissues when it comes to dynamic and complex surgical processes. In this paper, we propose SurgiSim, a novel automatic simulation system to overcome these limitations. To build a surgical simulation environment, we maintain a canonical 3D scene composed of 3D Gaussians coupled with a deformation field to accurately model monocular dynamic surgical scenes. This process involves a multi-stage optimization with trajectory and anisotropic regularization, enhancing the geometry consistency of the canonical scene which serves the simulation environment. To improve the realism of physical simulations, we implement a Visco-Elastic deformation model based on the Maxwell model, effectively restoring the complex deformations of tissues. Additionally, we estimate the physical properties of tissues by minimizing the discrepancies between the input video and simulation results guided by predicted tissue motion, ensuring realistic simulation outcomes. Experiments across diverse surgical scenarios demonstrate SurgiSim's ability to perform realistic physical interactions of soft tissues among surgical procedures, showing its enormous potential for enhancing surgical training, planning, and robotic surgery systems.
BibTeX
@misc{wang2025realistic,
  author = {Kailing Wang and Chen Yang and Keyang Zhao and Xiaokang Yang and Wei Shen},
  title = {{Realistic Surgical Simulation from Monocular Videos}},
  year = {2025},
  url = {https://openreview.net/forum?id=faSfhqDpZP},
  howpublished = {OpenReview},
  note = {Submitted to ICLR 2025},
}

Loading related content...

Figure