Abstract
Intraoperative imaging techniques for reconstructing deformable tissues in vivo are pivotal for advanced surgical systems. Existing methods either compromise on rendering quality or are excessively computationally intensive, often demanding dozens of hours to perform, which significantly hinders their practical application. In this paper, we introduce Fast Orthogonal Plane (Forplane), a novel, efficient framework based on neural radiance fields (NeRF) for the reconstruction of deformable tissues. We conceptualize surgical procedures as 4D volumes and break them down into static and dynamic fields comprised of orthogonal neural planes. This factorization discretizes the four-dimensional space, leading to decreased memory usage and faster optimization. A spatiotemporal importance sampling scheme is introduced to improve performance in regions with tool occlusion as well as large motions and accelerate training. An efficient ray marching method is applied to skip sampling among empty regions, significantly improving inference speed. Forplane accommodates both binocular and monocular endoscopy videos, demonstrating its extensive applicability and flexibility. Our experiments, carried out on two in vivo datasets, the EndoNeRF and Hamlyn datasets, demonstrate the effectiveness of our framework. In all cases, Forplane substantially accelerates both the optimization process (by over 100 times) and the inference process (by over 15 times) while maintaining or even improving the quality across a variety of non-rigid deformations.
Introduction
In robotic surgery, high-quality 3D reconstruction of tissues in real-time is a critical objective. It enables surgeons to navigate around vital structures like nerves and blood vessels and provides essential data for high-fidelity VR training. However, surgical environments present significant modeling challenges: tissues are constantly deforming, surgical tools frequently cause occlusions, and lighting conditions change dynamically with camera movement.
While Neural Radiance Fields (NeRF) have shown promise in producing high-fidelity results, they are typically computationally prohibitive. Previous state-of-the-art methods often require over 14 hours of training for a single short video clip, making them impractical for intraoperative use. The challenge lies in achieving high-fidelity, non-rigid reconstruction at speeds suitable for clinical application.
Methods: The Forplane Framework
The core innovation of this work is Forplane (Fast Orthogonal Plane), which replaces the traditional "heavy" Multi-Layer Perceptron (MLP) architecture of standard NeRF with a more efficient representation.
Orthogonal Neural Planes
Instead of treating the surgical scene as a continuous 4D volume ($X, Y, Z, Time$), Forplane factorizes the space into 2D planes. It utilizes three spatial planes ($XY, YZ, XZ$) to represent the static field (underlying anatomy) and three space-time planes ($XT, YT, ZT$) to represent the dynamic field (deformations). This factorization reduces computational complexity from $O(N^4)$ to $O(N^2)$, allowing for feature retrieval via fast bilinear interpolation.
Spatiotemporal Importance Sampling
To handle the complexities of surgical scenes, the model employs a spatiotemporal importance sampling scheme. Higher sampling probabilities are assigned to regions characterized by significant motion or tool occlusion. This forces the network to focus its learning capacity on the most challenging parts of the scene.
Efficient Ray Marching and Encoding
Forplane implements an "indicator grid" to track empty regions within the 4D space. The renderer skips these empty areas and only samples points near the tissue surface, significantly boosting inference speed. Additionally, the framework uses "one-blob" encoding instead of standard Fourier features. This better represents dynamic lighting conditions where the light source moves in tandem with the camera.
Monocular Support (MForplane)
Recognizing the prevalence of single-camera endoscopes, the authors introduced MForplane. This variant utilizes off-the-shelf depth predictors to provide geometric cues, enabling robust reconstruction even when stereo data is unavailable.
Experimental Evaluation
The researchers evaluated Forplane on two primary benchmarks:
- EndoNeRF Dataset: Six in-vivo video clips from DaVinci robotic prostatectomies, featuring tissue pulling, cutting, and tearing.
- Hamlyn Dataset: A public dataset with phantom heart and in-vivo sequences, known for weak textures and complex reflections.
Results
Forplane demonstrates a massive leap in performance across several metrics:
- Optimization Speed: High-quality results are achieved in just 10 minutes, a >100x improvement over EndoNeRF's 14+ hours.
- Inference Speed: The system reaches 1.73 fps, approximately 15 times faster than previous methods (0.11 fps).
- Reconstruction Quality: On the EndoNeRF dataset, Forplane achieved a PSNR of 37.31, significantly outperforming EndoNeRF (29.48) and TiNeuVox (31.60).
- Robustness: The "MForplane" variant maintained high fidelity with only a negligible drop in PSNR (approx. 1.8%) compared to the stereo version.
Conclusion and Future Work
Forplane represents a significant step toward real-time intraoperative tissue reconstruction. By factorizing 4D space into orthogonal planes and employing efficient sampling strategies, it achieves unprecedented speeds without sacrificing quality.
Future work will focus on automating tool segmentation to remove the reliance on manual masks and further optimizing ray marching to reach the 30 fps threshold required for seamless real-time display. Additionally, incorporating biomechanical properties and complex light-tissue interactions could further enhance the physical accuracy of the reconstructions.
Summary updated