Efficient Deformable Tissue Reconstruction via Orthogonal Neural Plane

Chen Yang, Kailing Wang, Yuehao Wang, Qi Dou, Xiaokang Yang, Wei Shen

TL;DR

Forplane (Fast Orthogonal Plane) is a new framework that reconstructs 3D deformable tissues from surgical endoscopy videos in minutes rather than hours. By decomposing 4D space-time into sets of 2D orthogonal planes, it achieves a 100x speedup in optimization and a 15x boost in inference speed compared to previous state-of-the-art methods like EndoNeRF, all while maintaining higher reconstruction quality.

What Problem Does This Paper Address?

In robotic surgery, having a high-quality 3D reconstruction of tissues in real-time is a "holy grail." It helps surgeons avoid critical structures like nerves and blood vessels and provides data for high-fidelity VR training.

However, surgical scenes are incredibly difficult to model: tissues deform constantly, surgical tools frequently block the view (occlusion), and lighting changes as the camera moves. Existing solutions based on Neural Radiance Fields (NeRF) produce beautiful results but are computationally "heavy," often requiring over 14 hours of training for a single short clip. This latency makes them impractical for actual intraoperative use. The challenge is to achieve high-fidelity, non-rigid reconstruction at speeds that approach real-time.

Methods

The core innovation is Forplane, which moves away from the traditional "heavy" MLP (Multi-Layer Perceptron) architecture used in NeRF.

  1. Orthogonal Neural Planes: Instead of treating the surgical scene as a continuous 4D volume ($X, Y, Z, Time$), Forplane factorizes the space into 2D planes. It uses three spatial planes ($XY, YZ, XZ$) to represent the static field (the underlying anatomy) and three space-time planes ($XT, YT, ZT$) to represent the dynamic field (the deformations). This reduces computational complexity from $O(N^4)$ to $O(N^2)$.
  2. Spatiotemporal Importance Sampling: The model doesn't treat every pixel equally. It assigns higher sampling probabilities to regions with significant motion or those occluded by tools, forcing the network to "study harder" on the most difficult parts of the scene.
  3. Efficient Ray Marching: The authors implement an "indicator grid" that tracks which parts of the 4D space are empty. This allows the renderer to skip empty space and only sample points near the tissue surface, drastically increasing inference speed.
  4. One-Blob Encoding: Standard NeRF uses Fourier features for positional encoding, which assumes a static light source. Since surgical lights move with the camera, Forplane uses "one-blob" encoding to better represent these dynamic lighting conditions.
  5. MForplane: The authors also introduce a variant for monocular (single-camera) endoscopes, using off-the-shelf depth predictors to provide geometric cues where stereo data is unavailable.

Datasets

The researchers evaluated Forplane on two primary benchmarks:

  • EndoNeRF Dataset: Consists of six in-vivo video clips from DaVinci robotic prostatectomies, featuring challenging scenarios like tissue pulling, cutting, and tearing.
  • Hamlyn Dataset: A public dataset containing both phantom heart and in-vivo sequences. It is known for having weak textures and complex reflections, providing a rigorous test for reconstruction robustness.

Key Insights and Ideas

  • Factorization is the Secret Sauce: By breaking the 4D problem into 2D planes, the model can use bilinear interpolation to fetch features. This is significantly faster than querying a deep neural network for every single point in space.
  • Static-Dynamic Disentanglement: By forcing the model to separate the scene into static and dynamic components via a specialized "disentangle loss," the system can "fill in the blanks" for tissue that is temporarily hidden behind a surgical tool.
  • Lighter is Better: Because the feature planes do most of the heavy lifting, Forplane can use a "tiny" MLP (only 2 layers) compared to the 8-layer MLPs typically used in NeRF. This is a major contributor to the inference speedup.

Key Results

  • Optimization Speed: Forplane reaches high-quality results in just 10 minutes, compared to over 14 hours for EndoNeRF (a >100x improvement).
  • Inference Speed: It achieves 1.73 fps, which is roughly 15 times faster than previous methods (0.11 fps). While not yet 30 fps, it is a massive leap toward real-time utility.
  • Accuracy: On the EndoNeRF dataset, Forplane achieved a PSNR of 37.31, significantly outperforming EndoNeRF (29.48) and TiNeuVox (31.60).
  • Monocular Performance: Even without stereo cameras, the "MForplane" variant maintained high fidelity, with only a negligible drop in PSNR (approx. 1.8%).

Limitations and Future Work

  • Tool Masking: The system currently relies on manually generated or pre-segmented masks to identify surgical tools. Future work aims to integrate automated, real-time tool tracking.
  • The "Real-Time" Gap: While 1.73 fps is a breakthrough, it is still below the threshold for seamless intraoperative display. Further optimization of the ray marching and grid structures is needed.
  • Complex Physics: The model focuses on visual reconstruction but does not yet account for the underlying biomechanical properties of the tissue or complex light-tissue interactions like sub-surface scattering.

Summary updated

Abstract

Intraoperative imaging techniques for reconstructing deformable tissues in vivo are pivotal for advanced surgical systems. Existing methods either compromise on rendering quality or are excessively computationally intensive, often demanding dozens of hours to perform, which significantly hinders their practical application. In this paper, we introduce Fast Orthogonal Plane (Forplane), a novel, efficient framework based on neural radiance fields (NeRF) for the reconstruction of deformable tissues. We conceptualize surgical procedures as 4D volumes, and break them down into static and dynamic fields comprised of orthogonal neural planes. This factorization discretizes the four-dimensional space, leading to a decreased memory usage and faster optimization. A spatiotemporal importance sampling scheme is introduced to improve performance in regions with tool occlusion as well as large motions and accelerate training. An efficient ray marching method is applied to skip sampling among empty regions, significantly improving inference speed. Forplane accommodates both binocular and monocular endoscopy videos, demonstrating its extensive applicability and flexibility. Our experiments, carried out on two in vivo datasets, the EndoNeRF and Hamlyn datasets, demonstrate the effectiveness of our framework. In all cases, Forplane substantially accelerates both the optimization process (by over 100 times) and the inference process (by over 15 times) while maintaining or even improving the quality across a variety of non-rigid deformations. This significant performance improvement promises to be a valuable asset for future intraoperative surgical applications. The code of our project is now available at https://github.com/Loping151/ForPlane .
BibTeX
@article{yang2024efficient,
  author = {Chen Yang and Kailing Wang and Yuehao Wang and Qi Dou and Xiaokang Yang and Wei Shen},
  title = {{Efficient Deformable Tissue Reconstruction via Orthogonal Neural Plane}},
  journal = {IEEE Trans. Medical Imaging},
  year = {2024},
  doi = {10.1109/tmi.2024.3388559},
}

Loading related content...

Figure