Shape-from-Shading under Near Point Lighting and Partial views for Orthopedic Endoscopy

Chenyu Wu, Srinivasa G. Narasimhan, Branislav Jaramaz2007

TL;DR

This paper introduces a novel shape-from-shading technique for reconstructing bone surfaces from endoscopic images. By combining shape-from-shading with multiple partial views and accounting for near-point lighting, it overcomes the limitations of traditional methods, enabling more accurate 3D visualization for minimally invasive orthopedic surgery.

What Problem Does This Paper Address?

Minimally invasive orthopedic surgery relies heavily on endoscopy for visualizing internal anatomy. However, reconstructing the 3D shape of bone from endoscopic images is challenging due to several factors:

  • Featureless Surfaces: Bones lack distinct textures, making traditional structure-from-motion techniques difficult.
  • Near-Point Lighting: Endoscopes use light sources very close to the camera, creating complex lighting conditions that deviate significantly from standard assumptions in computer vision.
  • Small Field of View & Partial Views: The limited view of an endoscope means only small portions of the bone are visible in any single image, and often only partial boundaries are captured.
  • Difficulty in Perception: These combined factors make it hard even for human experts to accurately infer bone shape from a single endoscopic image.

Existing methods often struggle with these specific challenges, necessitating a new approach for accurate and reliable 3D bone reconstruction in surgical settings.

Methods

The authors propose a two-pronged approach:

  1. Single-Image Shape-from-Shading (SFS) with Near-Point Lighting and Perspective Projection:
    • They extend classical SFS to handle the unique lighting conditions of endoscopes, where light sources are close to the camera and not necessarily co-located.
  2. This involves modeling the intensity fall-off from these near-point sources and incorporating perspective projection.
  3. Crucially, they also account for partial object boundaries in each image, which is a significant departure from SFS methods requiring complete contours.
  4. This allows for the estimation of both surface normals and depths from individual images.
  5. Global Shape-from-Shading using Multiple Partial Views:
    • To overcome the limitations of single-image reconstructions (due to partial views), they leverage sequences of endoscopic images.
  6. Endoscope Tracking: The endoscope's position and orientation are tracked in real-time to transform reconstructed shapes from local image coordinates to a global (world) coordinate frame.
  7. Iterative Alignment and Reconstruction: * An Iterative Closest Point (ICP) algorithm is used to align the partial shapes obtained from different views, improving the overall boundary matching.
  8. A "global shape-from-shading" process is then initiated. This involves simultaneously re-growing surface normals and depths across all views, guided by the aligned global contours. This iterative process refines the shape by propagating information between local views and global constraints.

Key Novelties:

  • Adaptation of SFS for near-point lighting and perspective projection in endoscopic settings.
  • Integration of partial boundary information into the SFS framework.
  • A global SFS approach that combines multiple partial views, leveraging ICP for alignment and an iterative refinement process for shape consistency.
  • A comprehensive calibration procedure for both geometric and photometric properties of the endoscope.

Datasets

Not discussed in the paper. The authors mention using simulations and experiments with artificial bones for evaluation.

Key Insights and Ideas

  • Near-point lighting is a game-changer: Standard SFS assumptions break down with close-up lighting. The paper's success hinges on accurately modeling this specific lighting model.
  • Partial boundaries are not a dead end: Instead of requiring complete contours, the method cleverly uses partial boundaries as starting points and then "grows" the shape, leveraging multiple views to fill in the gaps.
  • Global constraints are essential for consistency: While individual views provide local shape information, aligning them and then using global constraints during the SFS process is key to achieving a coherent and accurate overall reconstruction.
  • ICP is a crucial bridge: The ICP algorithm acts as a vital step to align the independently reconstructed partial shapes before the global refinement begins, mitigating errors from tracking and calibration.

Key Results

  • The proposed technique successfully reconstructs shapes from both simulated data and artificial bones.
  • For the artificial spine reconstruction, the accuracy achieved was within a practical range for surgery, with maximum, minimum, mean, and RMS errors of 3.1mm, 0.0mm, 1.16mm, and 1.5mm respectively, when compared to a laser range scanner ground truth.
  • The method demonstrates the ability to reconstruct shapes of objects like cylinders and parts of a spine, providing a more complete 3D representation than single-image SFS.
  • The global SFS process converges quickly, with an example of 18 images processed in approximately 5 minutes on a P4 2.4G CPU using Matlab.

Limitations and Future Work

  • Automatic Boundary Detection: The current method relies on manually labeled boundaries. Automating this process is a significant area for future work.
  • Initial Guesses: Obtaining good initial guesses for the shape reconstruction remains a challenge.
  • High-Frequency Details: Recovering fine, high-frequency details in the bone surface might be difficult with the current approach.
  • Image Noise: Dealing with noise from blood and tissues in real surgical environments is an open problem.
  • Real-time Performance: While the processing time is reasonable, achieving true real-time performance for immediate surgical feedback would be an advancement.
  • Low Resolution of Global Constraints: The paper notes that low resolution of global constraints can lead to aliasing, which can be mitigated by increasing resolution. This suggests a trade-off between computational cost and surface quality.

Summary updated

Abstract

Bone reconstruction using endoscopy is important for computer aided minimally invasive orthopedic surgery. During surgery an endoscope consisting of a camera and one or more light sources is inserted through a small inci-sion into the body and the acquired images are analyzed. Since bone surface is featureless, shading is the primary cue for shape perception. However, due to the small field of view of the endoscope, only a small part of the bone and its occluding contour are visible in any single image. There-fore even human perception of bone shape from such images can be hard. We present a novel technique to reconstruct the surface of the bone by applying shape-from-shading to a sequence of endoscopic images, with partial boundary in each image. We first perform geometric and the photometric calibration for the endoscope. We then extend the classical shape-from-shading algorithm to include a near point light source that is not optically co-located with the camera. By tracking the endoscope we are able to align partial shapes obtained from different images in the global (world) coordinates. An ICP algorithm is then used to improve the matching, result-ing in a complete occluding boundary of the bone. Finally, a complete and consistent shape is obtained by simultane-ously re-growing surface normals and depths in all views. We demonstrate the accuracy of our technique using simu-lations and experiments with artificial bones. 1.
BibTeX
@misc{wu2007shape,
  author = {Chenyu Wu and Srinivasa G. Narasimhan and Branislav Jaramaz},
  title = {{Shape-from-Shading under Near Point Lighting and Partial views for Orthopedic Endoscopy}},
  year = {2007},
}
Figure