Introduction
Knee arthroscopy is a cornerstone of minimally invasive orthopedic surgery, allowing clinicians to diagnose and treat joint issues through small incisions using an endoscope. However, the intra-operative environment presents significant challenges for computer vision. The field of view is narrow, and images are frequently degraded by artifacts such as overexposure from the light source, motion blur, and a lack of distinct surface texture. Furthermore, when the scope is positioned too close to tissue, the resulting images become highly distorted or out of focus.
Automatic semantic segmentation—the task of identifying anatomical structures like the femur, tibia, meniscus, and anterior cruciate ligament (ACL)—could significantly aid surgeon orientation and navigation. Despite its potential, training robust deep learning models for this task is difficult due to the scarcity of annotated surgical data and the tendency of models to overfit on the noisy, low-quality textures found in arthroscopic video.
Methodology
This paper proposes a multi-task learning (MTL) framework that leverages self-supervised depth estimation to "regularise" the semantic segmentation task. The core hypothesis is that forcing a model to understand the 3D geometry of the scene prevents it from overfitting to misleading 2D visual artifacts.
The authors utilize a modified UNet++ architecture and implement a novel two-stage training pipeline:
- Stage 1: Pre-training for Depth: The model's encoder and a depth decoder are first trained on a dataset of "clean" stereo images of everyday objects. These images provide rich geometric information without surgical noise. Using a self-supervised loss based on stereo disparity, the model learns the fundamentals of monocular depth estimation.
- Stage 2: Fine-tuning with Multi-Task Learning: The pre-trained encoder is then transferred to the surgical domain. Here, the model is fine-tuned to perform two tasks simultaneously: supervised semantic segmentation (using manual labels) and self-supervised depth estimation (using stereo pairs from the surgical site).
By sharing the encoder between these two tasks, the depth estimation acts as a geometric guardrail, ensuring that the features extracted for segmentation are grounded in the physical 3D structure of the knee joint.
Experimental Setup and Datasets
The study utilized three distinct datasets to facilitate this training regime:
- Pre-training Set: 2,150 stereo image pairs of routine, non-surgical objects.
- Segmentation Set: 3,868 arthroscopic images captured from four cadaveric specimens, manually annotated for four classes: Femur, ACL, Tibia, and Meniscus.
- Fine-tuning Depth Set: 2,000 stereo image pairs captured during the same cadaveric procedures to support the self-supervised depth task.
Key Results
The proposed depth-regularised model demonstrated superior performance over several baselines. It achieved a mean Dice score of 0.603, significantly outperforming a standard UNet++ (0.560) and a previous state-of-the-art joint learning approach (0.595).
One of the most notable improvements was in the segmentation of the Anterior Cruciate Ligament (ACL). The ACL is notoriously difficult to identify due to its variable appearance and frequent occlusion, yet the inclusion of depth information led to substantial gains in its identification. Additionally, the authors confirmed the necessity of the pre-training stage; when Stage 1 was skipped, the mean Dice score dropped from 0.603 to 0.573, highlighting the importance of establishing a geometric prior on clean data before tackling messy surgical images.
Discussion and Future Work
The research demonstrates that self-supervised depth estimation is an effective regulariser for surgical segmentation, especially when ground-truth depth data is unavailable. The model is also computationally efficient, processing single images in approximately 50ms, which meets the requirements for real-time intra-operative assistance.
However, limitations remain. The study relied on cadaveric data, which lacks the blood and active fluid flow present in live surgery. Furthermore, while the femur is segmented with high accuracy, performance on the tibia and meniscus remains lower. Future research will focus on refining the segmentation of these complex structures and validating the framework in live clinical environments.
Summary updated