Abstract
Traditional computer-assisted orthopaedic surgery (CAOS) relies on invasive infrared markers pinned directly into the bone. This paper introduces a robust, markerless bone-tracking system that utilizes RGB-D cameras and deep learning to eliminate the need for these physical markers. By processing data as 3D point clouds and training on a custom synthetic dataset with simulated occlusions, the system maintains high accuracy even when the camera's view is partially obstructed by a surgeon's hands or tools. The proposed pipeline achieves clinically acceptable accuracy and operates in real-time.
Introduction
In current orthopaedic procedures, "gold standard" tracking involves pinning physical markers into the patient's bone. While accurate, this approach is invasive, increases surgical time, and requires a constant line-of-sight to an optical tracker. Markerless alternatives often struggle in the chaotic surgical environment where hands, gauze, and instruments frequently occlude the bone. Furthermore, many existing models are tied to specific hardware. This research addresses these challenges by developing a tracking system that is robust to occlusion and adaptable across different camera hardwares.
Methods
The proposed system employs a two-stage deep learning pipeline to estimate bone pose from RGB-D data.
System Architecture
The pipeline consists of three main components:
- RoINet (Region of Interest Prediction): An RGB-based network that identifies the bone's general location. Unlike fixed-crop methods, RoINet predicts an adaptive bounding box, making the system camera-agnostic by handling various focal lengths and distances.
- SegNet (3D Segmentation): The depth data within the predicted RoI is converted into a 3D point cloud. A PointNet-based architecture classifies points as bone or distractors (e.g., hands, tools).
- BICP (Bounded Iterative Closest Point): The segmented bone points are registered to a pre-scanned 3D model of the patient's bone to determine its exact pose.
Synthetic Data Pipeline
To overcome the difficulty of labeling real-world surgical data, the authors developed a synthetic data pipeline using BlenderProc. They generated 10,000 randomized RGB-D images, simulating various lighting conditions, backgrounds, and realistic sensor noise. Crucially, the dataset includes 3D models of hands and tools to simulate foreground occlusions.
Evaluation
The model was evaluated on a real-world test set of 800 captures using an Intel RealSense D415. Ground truth for occluded images was obtained via a "pairwise capture" method, filming the bone both with and without obstructions from a fixed camera position. Generalization was further tested on a cadaver knee and an alternative RGB-D camera.
Results and Discussion
The system demonstrates significant resilience to occlusion and hardware variation.
- Accuracy: Without occlusion, the system achieved a rotational accuracy of ~1.02° and translational accuracy of ~4.39 mm. Under heavy occlusion, accuracy remained stable (~2.05° and ~4.33 mm), staying within the clinically acceptable range for many procedures.
- 3D Robustness: Learning features in 3D space proved superior to 2D depth maps, as the network could effectively ignore depth dropout and sensor artifacts.
- Performance: The pipeline runs at approximately 12 Hz on a standard laptop CPU, outperforming previous state-of-the-art methods in segmentation accuracy (IoU).
Conclusion
This work presents a viable markerless alternative to invasive physical tracking in orthopaedic surgery. By leveraging synthetic data and 3D point cloud processing, the system achieves the robustness necessary for real-world clinical environments. Future work will focus on increasing the update rate to 30 Hz and addressing non-rigid body deformations of surrounding soft tissue.
Summary updated