Occlusion-robust Visual Markerless Bone Tracking for Computer-Assisted Orthopaedic Surgery

Xue Hu, Anh Nguyen, Ferdinando Rodriguez y Baena

Abstract

Traditional computer-assisted orthopaedic surgery (CAOS) relies on invasive infrared markers pinned directly into the bone. This paper introduces a robust, markerless bone-tracking system that utilizes RGB-D cameras and deep learning to eliminate the need for these physical markers. By processing data as 3D point clouds and training on a custom synthetic dataset with simulated occlusions, the system maintains high accuracy even when the camera's view is partially obstructed by a surgeon's hands or tools. The proposed pipeline achieves clinically acceptable accuracy and operates in real-time.

Introduction

In current orthopaedic procedures, "gold standard" tracking involves pinning physical markers into the patient's bone. While accurate, this approach is invasive, increases surgical time, and requires a constant line-of-sight to an optical tracker. Markerless alternatives often struggle in the chaotic surgical environment where hands, gauze, and instruments frequently occlude the bone. Furthermore, many existing models are tied to specific hardware. This research addresses these challenges by developing a tracking system that is robust to occlusion and adaptable across different camera hardwares.

Methods

The proposed system employs a two-stage deep learning pipeline to estimate bone pose from RGB-D data.

System Architecture

The pipeline consists of three main components:

  1. RoINet (Region of Interest Prediction): An RGB-based network that identifies the bone's general location. Unlike fixed-crop methods, RoINet predicts an adaptive bounding box, making the system camera-agnostic by handling various focal lengths and distances.
  2. SegNet (3D Segmentation): The depth data within the predicted RoI is converted into a 3D point cloud. A PointNet-based architecture classifies points as bone or distractors (e.g., hands, tools).
  3. BICP (Bounded Iterative Closest Point): The segmented bone points are registered to a pre-scanned 3D model of the patient's bone to determine its exact pose.
Overview of the markerless segmentation and registration pipeline.

Synthetic Data Pipeline

To overcome the difficulty of labeling real-world surgical data, the authors developed a synthetic data pipeline using BlenderProc. They generated 10,000 randomized RGB-D images, simulating various lighting conditions, backgrounds, and realistic sensor noise. Crucially, the dataset includes 3D models of hands and tools to simulate foreground occlusions.

Synthetic data generation examples with simulated lighting and occlusions.

Evaluation

The model was evaluated on a real-world test set of 800 captures using an Intel RealSense D415. Ground truth for occluded images was obtained via a "pairwise capture" method, filming the bone both with and without obstructions from a fixed camera position. Generalization was further tested on a cadaver knee and an alternative RGB-D camera.

Results and Discussion

The system demonstrates significant resilience to occlusion and hardware variation.

  • Accuracy: Without occlusion, the system achieved a rotational accuracy of ~1.02° and translational accuracy of ~4.39 mm. Under heavy occlusion, accuracy remained stable (~2.05° and ~4.33 mm), staying within the clinically acceptable range for many procedures.
  • 3D Robustness: Learning features in 3D space proved superior to 2D depth maps, as the network could effectively ignore depth dropout and sensor artifacts.
  • Performance: The pipeline runs at approximately 12 Hz on a standard laptop CPU, outperforming previous state-of-the-art methods in segmentation accuracy (IoU).

Conclusion

This work presents a viable markerless alternative to invasive physical tracking in orthopaedic surgery. By leveraging synthetic data and 3D point cloud processing, the system achieves the robustness necessary for real-world clinical environments. Future work will focus on increasing the update rate to 30 Hz and addressing non-rigid body deformations of surrounding soft tissue.

Summary updated

Abstract

Conventional computer-assisted orthopaedic navigation systems rely on the tracking of dedicated optical markers for patient poses, which makes the surgical workflow more invasive, tedious, and expensive. Visual tracking has recently been proposed to measure the target anatomy in a markerless and effortless way, but the existing methods fail under real-world occlusion caused by intraoperative interventions. Furthermore, such methods are hardware-specific and not accurate enough for surgical applications. In this paper, we propose a RGB-D sensing-based markerless tracking method that is robust against occlusion. We design a new segmentation network that features dynamic region-of-interest prediction and robust 3D point cloud segmentation. As it is expensive to collect large-scale training data with occlusion instances, we also propose a new method to create synthetic RGB-D images for network training. Experimental results show that our proposed markerless tracking method outperforms recent state-of-the-art approaches by a large margin, especially when an occlusion exists. Furthermore, our method generalises well to new cameras and new target models, including a cadaver, without the need for network retraining. In practice, by using a high-quality commercial RGB-D camera, our proposed visual tracking method achieves an accuracy of 1-2 degress and 2-4 mm on a model knee, which meets the standard for clinical applications.
BibTeX
@article{hu2022occlusion,
  author = {Xue Hu and Anh Nguyen and Ferdinando Rodriguez y Baena},
  title = {{Occlusion-robust Visual Markerless Bone Tracking for Computer-Assisted Orthopaedic Surgery}},
  journal = {IEEE Trans. Instrumentation \& Measurement},
  year = {2022},
  url = {https://openreview.net/forum?id=4pf9WY5biAR},
}
Figure