Self-supervised Depth Estimation to Regularise Semantic Segmentation in Knee Arthroscopy

Fengbei Liu, Yaqub Jonmohamadi, Gabriel Maicas, Ajay K. Pandey, Gustavo Carneiro

Introduction

Knee arthroscopy is a cornerstone of minimally invasive orthopedic surgery, allowing clinicians to diagnose and treat joint issues through small incisions using an endoscope. However, the intra-operative environment presents significant challenges for computer vision. The field of view is narrow, and images are frequently degraded by artifacts such as overexposure from the light source, motion blur, and a lack of distinct surface texture. Furthermore, when the scope is positioned too close to tissue, the resulting images become highly distorted or out of focus.

Common visual artifacts in knee arthroscopy images, including overexposure, poor texture, and focus issues.

Automatic semantic segmentation—the task of identifying anatomical structures like the femur, tibia, meniscus, and anterior cruciate ligament (ACL)—could significantly aid surgeon orientation and navigation. Despite its potential, training robust deep learning models for this task is difficult due to the scarcity of annotated surgical data and the tendency of models to overfit on the noisy, low-quality textures found in arthroscopic video.

Methodology

This paper proposes a multi-task learning (MTL) framework that leverages self-supervised depth estimation to "regularise" the semantic segmentation task. The core hypothesis is that forcing a model to understand the 3D geometry of the scene prevents it from overfitting to misleading 2D visual artifacts.

The authors utilize a modified UNet++ architecture and implement a novel two-stage training pipeline:

  1. Stage 1: Pre-training for Depth: The model's encoder and a depth decoder are first trained on a dataset of "clean" stereo images of everyday objects. These images provide rich geometric information without surgical noise. Using a self-supervised loss based on stereo disparity, the model learns the fundamentals of monocular depth estimation.
  2. Stage 2: Fine-tuning with Multi-Task Learning: The pre-trained encoder is then transferred to the surgical domain. Here, the model is fine-tuned to perform two tasks simultaneously: supervised semantic segmentation (using manual labels) and self-supervised depth estimation (using stereo pairs from the surgical site).
The proposed two-stage training pipeline: pre-training on routine objects followed by multi-task fine-tuning on arthroscopic data.

By sharing the encoder between these two tasks, the depth estimation acts as a geometric guardrail, ensuring that the features extracted for segmentation are grounded in the physical 3D structure of the knee joint.

Experimental Setup and Datasets

The study utilized three distinct datasets to facilitate this training regime:

  • Pre-training Set: 2,150 stereo image pairs of routine, non-surgical objects.
  • Segmentation Set: 3,868 arthroscopic images captured from four cadaveric specimens, manually annotated for four classes: Femur, ACL, Tibia, and Meniscus.
  • Fine-tuning Depth Set: 2,000 stereo image pairs captured during the same cadaveric procedures to support the self-supervised depth task.

Key Results

The proposed depth-regularised model demonstrated superior performance over several baselines. It achieved a mean Dice score of 0.603, significantly outperforming a standard UNet++ (0.560) and a previous state-of-the-art joint learning approach (0.595).

One of the most notable improvements was in the segmentation of the Anterior Cruciate Ligament (ACL). The ACL is notoriously difficult to identify due to its variable appearance and frequent occlusion, yet the inclusion of depth information led to substantial gains in its identification. Additionally, the authors confirmed the necessity of the pre-training stage; when Stage 1 was skipped, the mean Dice score dropped from 0.603 to 0.573, highlighting the importance of establishing a geometric prior on clean data before tackling messy surgical images.

Discussion and Future Work

The research demonstrates that self-supervised depth estimation is an effective regulariser for surgical segmentation, especially when ground-truth depth data is unavailable. The model is also computationally efficient, processing single images in approximately 50ms, which meets the requirements for real-time intra-operative assistance.

However, limitations remain. The study relied on cadaveric data, which lacks the blood and active fluid flow present in live surgery. Furthermore, while the femur is segmented with high accuracy, performance on the tibia and meniscus remains lower. Future research will focus on refining the segmentation of these complex structures and validating the framework in live clinical environments.

Summary updated

Abstract

Intra-operative automatic semantic segmentation of knee joint structures can assist surgeons during knee arthroscopy in terms of situational awareness. However, due to poor imaging conditions (e.g., low texture, overexposure, etc.), automatic semantic segmentation is a challenging scenario, which justifies the scarce literature on this topic. In this paper, we propose a novel self-supervised monocular depth estimation to regularise the training of the semantic segmentation in knee arthroscopy. To further regularise the depth estimation, we propose the use of clean training images captured by the stereo arthroscope of routine objects (presenting none of the poor imaging conditions and with rich texture information) to pre-train the model. We fine-tune such model to produce both the semantic segmentation and self-supervised monocular depth using stereo arthroscopic images taken from inside the knee. Using a data set containing 3868 arthroscopic images captured during cadaveric knee arthroscopy with semantic segmentation annotations, 2000 stereo image pairs of cadaveric knee arthroscopy, and 2150 stereo image pairs of routine objects, we show that our semantic segmentation regularised by self-supervised depth estimation produces a more accurate segmentation than a state-of-the-art semantic segmentation approach modeled exclusively with semantic segmentation annotation.
BibTeX
@inproceedings{liu2020self,
  author = {Fengbei Liu and Yaqub Jonmohamadi and Gabriel Maicas and Ajay K. Pandey and Gustavo Carneiro},
  title = {{Self-supervised Depth Estimation to Regularise Semantic Segmentation in Knee Arthroscopy}},
  booktitle = {International Conference on Medical Image Computing and Computer-Assisted Intervention},
  year = {2020},
  url = {https://openreview.net/forum?id=RWsLDzcQJZA},
}
Figure