EndoDAC: Efficient Adapting Foundation Model for Self-Supervised Depth Estimation from Any Endoscopic Camera

Beilei Cui, Mobarakol Islam, Long Bai, An Wang, Hongliang Ren

Abstract

Accurate depth estimation is essential for surgical navigation and augmented reality in minimally invasive procedures. However, surgical environments present unique challenges, including complex lighting, specular reflections, and a lack of ground truth labels. We introduce EndoDAC (Endoscopic Depth Any Camera), a framework that adapts vision foundation models to the surgical domain using a novel Dynamic Vector-Based Low-Rank Adaptation (DV-LoRA) and a self-supervised "any camera" learning strategy. EndoDAC achieves state-of-the-art performance on standard benchmarks with minimal trainable parameters and real-time inference speeds.

Introduction

In minimally invasive surgery, depth perception is vital for safe navigation and the integration of augmented reality. Standard computer vision models often struggle in surgical environments due to dim lighting, fluid reflections, and low-texture surfaces. While large-scale vision foundation models (VFMs) have shown remarkable generalization in natural scenes, their direct application to surgery is limited by the significant domain gap and the high cost of full-parameter fine-tuning. Moreover, most existing depth estimation methods require precise camera intrinsic parameters, which are frequently unavailable in clinical practice.

Methodology

EndoDAC addresses these challenges through three primary architectural innovations designed to bridge the gap between general-purpose foundation models and specialized surgical needs.

Overview of the EndoDAC architecture, featuring the DepthNet for depth estimation and the Pose-Intrinsics Net for camera parameter estimation.
  1. DV-LoRA (Dynamic Vector-Based Low-Rank Adaptation): To bridge the domain gap efficiently, we utilize DV-LoRA. This method initially trains small adapter layers and subsequently transitions to even more compact dynamic vectors. This strategy allows the model to adapt to surgical scenes using only 1.6 million trainable parameters—just 1.6% of the total model size—significantly reducing the computational burden of fine-tuning large VFMs.
  2. Convolutional Neck Blocks: While Vision Transformers (ViTs) excel at capturing global context, they often lose high-frequency spatial details. We insert CNN-based "neck" blocks between transformer stages to recover these sharp details, which are critical for precise surgical depth estimation.
  3. Self-Supervised "Any Camera" Learning: To eliminate the need for manual camera calibration, we incorporate a Pose-Intrinsics Net. This sub-network simultaneously estimates camera motion and internal parameters (intrinsics) by analyzing temporal consistency across video frames. This allows EndoDAC to learn from uncalibrated monocular endoscopic videos in a self-supervised manner.

Experiments and Results

We evaluated EndoDAC on the SCARED (porcine cadaver anatomy) and Hamlyn (in vivo laparoscopic video) datasets. EndoDAC achieved an Absolute Relative Error (Abs Rel) of 0.051 on the SCARED dataset, outperforming the previous state-of-the-art self-supervised method (0.058).

Qualitative comparison of depth maps produced by EndoDAC versus other state-of-the-art methods.

Qualitative results highlight the model's ability to produce smoother and more accurate depth maps compared to zero-shot foundation models and specialized surgical depth estimators. The model maintains an inference speed of 17.7 ms per frame, supporting real-time surgical applications. Furthermore, its performance on the Hamlyn dataset after training on SCARED demonstrates robust zero-shot generalization across different surgical procedures and camera types.

Conclusion

EndoDAC presents a highly efficient and flexible approach to surgical depth estimation. By leveraging the strengths of foundation models through lightweight adaptation and learning camera intrinsics on the fly, it overcomes major barriers to clinical deployment. Future work will focus on further model distillation for low-power medical hardware and testing on a broader range of specialized endoscopic instruments.

Summary updated

BibTeX
@inproceedings{cui2024endodac,
  author = {Beilei Cui and Mobarakol Islam and Long Bai and An Wang and Hongliang Ren},
  title = {{EndoDAC: Efficient Adapting Foundation Model for Self-Supervised Depth Estimation from Any Endoscopic Camera}},
  booktitle = {International Conference on Medical Image Computing and Computer-Assisted Intervention},
  year = {2024},
  doi = {10.1007/978-3-031-72089-5\_20},
}
Figure