Deform3DGS: Flexible Deformation for Fast Surgical Scene Reconstruction with Gaussian Splatting

Shuojue Yang, Qian Li, Daiyun Shen, Bingchen Gong, Qi Dou, Yueming Jin

Abstract

Accurate 3D reconstruction of dynamic surgical scenes is essential for robotic surgery and intraoperative navigation. However, existing methods often struggle with the computational demands of real-time performance and the complexities of tissue deformation and tool occlusion. We present Deform3DGS, a framework utilizing 3D Gaussian Splatting and a novel Flexible Deformation Modeling (FDM) approach. By employing learnable Gaussian basis functions for temporal modeling and a Motion-Aware Point Fusion (MAPF) strategy for initialization, Deform3DGS achieves state-of-the-art reconstruction quality. Notably, it enables rendering at over 330 FPS and reduces training time to approximately one minute, significantly outperforming previous NeRF-based and Gaussian-based solutions.

Introduction

Accurate 3D reconstruction of surgical scenes is vital for intraoperative navigation and robotic surgery. However, the surgical environment presents two massive hurdles: tissue deformation and occlusion. Tissues are constantly shifting due to heartbeat, respiration, and instrument interaction. Furthermore, surgical tools frequently block the camera's view of the underlying anatomy.

Existing solutions, largely based on Neural Radiance Fields (NeRF), produce high-quality results but are notoriously slow. They often require hours of training and offer sluggish rendering speeds, which is a dealbreaker for live clinical applications. While some recent "fast" methods have emerged, they often compromise on reconstruction fidelity or still rely on computationally expensive Multi-Layer Perceptrons (MLPs) to handle motion.

Methodology

Deform3DGS replaces the implicit representations of NeRF with the explicit, point-based representation of 3D Gaussian Splatting (3DGS). The framework consists of three main pillars:

  1. Motion-Aware Point Fusion (MAPF): Standard 3DGS often starts with a sparse point cloud. In surgery, tool occlusions create "voids" in the initial data. MAPF identifies regions with significant motion or missing data across frames and fuses points from multiple timestamps to create a dense, robust initialization for the canonical scene.
  2. Flexible Deformation Modeling (FDM): Instead of using a heavy MLP to predict how each Gaussian moves over time, FDM uses a linear combination of learnable Gaussian basis functions. Each basis function has its own learnable center and variance, allowing the model to adapt specifically to the local temporal dynamics of the tissue.
  3. Efficient Attribute Regression: The FDM predicts changes in position, rotation, and scale for every individual Gaussian. Because the basis functions are localized in time, the model can capture intricate, non-linear deformations (like a tool prodding tissue) without affecting the entire timeline.
The Deform3DGS pipeline, showing point cloud initialization (MAPF), flexible deformation modeling (FDM), and the final splatting process.

Experiments and Results

The model was evaluated on two prominent surgical datasets: EndoNeRF (robotic prostatectomy) and StereoMIS (in-vivo porcine surgery). These datasets feature complex instrument occlusions and diverse anatomical structures.

Deform3DGS sets a new bar for speed-to-quality ratio in surgical reconstruction:

  • Training Speed: It reduces training time from hours (EndoNeRF) or minutes (EndoGaussian) to just ~64 seconds.
  • Rendering Performance: It achieves 338.8 FPS on the EndoNeRF dataset, more than double the speed of the previous state-of-the-art Gaussian-based method.
  • Reconstruction Quality: Despite the speedup, it maintains or exceeds SOTA fidelity, reaching a PSNR of 37.90 on EndoNeRF.
Quantitative comparison of Deform3DGS against SOTA methods on EndoNeRF and StereoMIS datasets.

Key Insights

  • Temporal Locality: Traditional explicit models often use fixed global bases. By using learnable Gaussian kernels, Deform3DGS achieves temporal locality—it can model a specific interaction at one timestamp without affecting the reconstruction at others.
  • Initialization and Efficiency: Motion-aware dense initialization (MAPF) significantly stabilizes training. Furthermore, formulating deformation as a linear combination of basis functions rather than an MLP inference drastically reduces computational overhead during both training and rendering.

Conclusion and Future Work

Deform3DGS demonstrates that high-fidelity, real-time surgical reconstruction is achievable within a one-minute training window. While currently limited by sequence length and a dependency on stereo depth maps, future work will focus on scaling to longer sequences and adapting the framework for monocular endoscopic data. Explicitly modeling surgical tools as separate entities could also further improve scene understanding.

Summary updated

BibTeX
@inproceedings{yang2024deform,
  author = {Shuojue Yang and Qian Li and Daiyun Shen and Bingchen Gong and Qi Dou and Yueming Jin},
  title = {{Deform3DGS: Flexible Deformation for Fast Surgical Scene Reconstruction with Gaussian Splatting}},
  booktitle = {International Conference on Medical Image Computing and Computer-Assisted Intervention},
  year = {2024},
  doi = {10.1007/978-3-031-72089-5\_13},
}

Loading related content...

Figure