Abstract
This paper tackles the challenge of automatically performing realistic surgical simulations from readily available surgical videos. Recent efforts have successfully integrated physically grounded dynamics within 3D Gaussians to perform high-fidelity simulations in well-reconstructed simulation environments from static scenes. However, they struggle with the geometric inconsistency in reconstructing simulation environments and unrealistic physical deformations in simulations of soft tissues when it comes to dynamic and complex surgical processes. In this paper, we propose SurgiSim, a novel automatic simulation system to overcome these limitations. To build a surgical simulation environment, we maintain a canonical 3D scene composed of 3D Gaussians coupled with a deformation field to represent a dynamic surgical scene. This process involves a multi-stage optimization with trajectory and anisotropic regularization, enhancing the geometry consistency of the canonical scene, which serves as the simulation environment. To achieve realistic physical simulations in this environment, we implement a Visco-Elastic deformation model based on the Maxwell model, effectively restoring the complex deformations of tissues. Additionally, we infer the physical parameters of tissues by minimizing the discrepancies between the input video and simulation results guided by estimated tissue motion, ensuring realistic simulation outcomes. Experiments on various surgical scenarios and interactions demonstrate SurgiSim's ability to perform realistic simulation of soft tissues among surgical procedures, showing its enormous potential for enhancing surgical training, planning, and robotic surgery systems.
Introduction
Realistic surgical simulation is vital for training surgeons and developing robotic systems, yet creating these environments is traditionally a bottleneck. Current mainstream systems rely on manual mesh-based modeling and tedious parameter tuning. While recent AI advancements like Neural Radiance Fields (NeRF) and 3D Gaussian Splatting (3DGS) have made strides in reconstructing scenes from video, they face two major hurdles in a surgical context: geometric inconsistency and unrealistic physics.
Most reconstruction methods focus on visual "plausibility" per frame, which often leads to underlying geometry that fragments during deformation. Furthermore, existing physics-integrated models often use simple elastic models, making organs look like "jelly" that oscillates indefinitely. In reality, human tissue is visco-elastic—it deforms and then quickly settles due to internal damping.
Methodology
SurgiSim introduces an automated pipeline that transforms standard monocular surgical videos into high-fidelity 3D simulation environments through a multi-stage approach:
- Canonical Scene Reconstruction: The system uses 3DGS to represent the scene. To ensure the geometry remains "simulation-ready," it employs a multi-stage optimization strategy including trajectory regularization (ensuring neighboring points move in parallel) and anisotropic regularization (preventing Gaussians from becoming too thin).
- Surface Thickening: Since monocular video only captures the surface of an organ, SurgiSim pushes Gaussian kernels along the z-axis to give the reconstructed surface the volume and thickness required for physical simulation.
- Visco-Elastic Modeling: SurgiSim implements a Maxwell-based visco-elastic model. This incorporates a viscous component that handles energy dissipation, allowing the simulated tissue to rebound realistically without excessive shaking.
- Differentiable Parameter Estimation: The system "learns" physical properties (like Young’s modulus and viscosity) directly from the video. It uses point tracking and depth estimation to observe tissue motion, then employs a differentiable Material Point Method (MPM) to optimize parameters until the simulated motion matches the source footage.
Experimental Results
The system was evaluated using the EndoNeRF dataset. Notably, SurgiSim only requires monocular (left-eye) footage, making it applicable to a wide range of existing surgical data where stereo information might be missing.
SurgiSim outperformed baselines across standard metrics, achieving a PSNR of 22.455 and SSIM of 0.7315. In a user study of 68 participants, 57.8% of surgeons preferred SurgiSim’s results, specifically noting the realistic "rebound" behavior of the tissue compared to the "jelly-like" oscillations of simpler models.
Conclusion and Future Work
SurgiSim demonstrates that treating input video as a "physics lab" allows for the bridge between computer vision and realistic graphics. By enforcing trajectory consistency and utilizing visco-elastic damping, it achieves a new level of realism for soft-tissue interaction.
Future work aims to address current limitations, such as reconstructing the "back side" of tissues not visible to the camera and improving the visual realism of newly exposed interior tissue during cutting procedures. Leveraging Large Reconstruction Models (LRMs) could eventually allow the system to "hallucinate" these missing internal structures automatically.
Summary updated