TL;DR
This paper tackles the challenge of monocular depth estimation in surgical videos, where large datasets are scarce. They demonstrate that pre-trained natural image depth models can be effectively transferred to the surgical domain, significantly improved by a novel temporal consistency self-supervision technique. This approach outperforms existing methods and establishes a new benchmark dataset for future research.
What Problem Does This Paper Address?
Estimating depth from a single camera image (monocular depth estimation) is a well-studied problem in computer vision, with deep learning models achieving impressive results on natural images. However, applying these models to surgical videos presents a significant challenge. Surgical datasets with accurate ground truth depth are rare and small, making it difficult to train models from scratch for this domain. Existing state-of-the-art models trained on massive natural image datasets often struggle with the unique visual characteristics and potential temporal inconsistencies found in surgical footage. This paper aims to bridge this gap by investigating how to effectively transfer and adapt these powerful natural image models for surgical vision tasks, addressing the limitations of low-data regimes and temporal instability.
Methods
The core of the paper's methodology involves two main strategies:
- Transfer Learning with Meta-Datasets: The authors leverage large, pre-trained monocular depth models (like MiDaS and Depth Anything) that were trained on vast natural image datasets. They then fine-tune these models on a newly curated surgical meta-dataset called Meta-MED. This approach capitalizes on the general geometric understanding learned from natural images.
- Temporal Consistency Self-Supervision: To further improve performance and address temporal inconsistencies in surgical videos, they introduce a novel self-supervision loss, $L_{temp}$. This loss encourages the model to produce consistent depth predictions for temporally adjacent frames. It works by inferring depth maps for two close frames, calculating optical flow between them, and then warping one depth map to align with the other. The difference between the warped and original depth maps, after masking, forms the basis of the loss. This is implemented using a "slow/teacher" and "fast/student" model architecture to prevent trivial solutions. They also explore augmentation consistency ($L_{aug}$) as another self-supervision technique.
- Scale and Shift Invariant Mean Absolute Error (SSIMAE): For evaluation and loss calculation, they use a metric that is invariant to scale and shift differences between predicted and ground truth depth maps. This is crucial for relative depth estimation, where the absolute scale is not always known or consistent.
Datasets
- Meta-MED: The authors introduce Meta-MED, a new meta-dataset specifically designed for training and evaluating relative monocular depth models in the endoscopic domain. It's an ensemble of existing surgical depth datasets.
- SCARED [2] and SERV-CT [8]: These are two high-quality datasets used for dense depth estimation in minimally invasive surgery. SCARED uses structured light projection, while SERV-CT uses pre-operative CT scans. These are combined to form a holdout testing dataset due to their small size.
- RobutsMIS [22] and Cholec80 [27]: These datasets, totaling 78 hours of footage from 110 in-vivo human surgeries, are used for self-supervised training due to their large volume and dynamic nature, despite lacking ground truth depth.
Key Insights and Ideas
- Transferability is Strong: Pre-trained natural image depth models exhibit a surprising degree of generalizability to the surgical domain, even before fine-tuning. This suggests that fundamental geometric understanding is largely transferable.
- Temporal Consistency is Crucial for Surgery: Surgical videos are inherently dynamic. The proposed temporal consistency self-supervision significantly improves performance, especially in addressing the temporal instability that plagues other methods.
- Self-Supervision Outperforms Standard Fine-tuning: When dealing with the low-data regime of surgical videos, self-supervision (particularly temporal consistency) provides a much larger performance boost than standard supervised fine-tuning alone.
- Meta-Dataset Approach is Effective: Combining multiple smaller surgical datasets into a meta-dataset (Meta-MED) creates a more robust training resource than any single dataset alone, leading to better generalizable models.
Key Results
- Significant Improvement: Fine-tuning pre-trained models on Meta-MED, especially with the addition of temporal consistency self-supervision, leads to substantial improvements in depth estimation accuracy (measured by SSIMAE) on surgical datasets.
- Outperforming State-of-the-Art: The authors' best models, particularly those utilizing temporal consistency, significantly outperform the domain-specific state-of-the-art method, AF-SFMLearner. For example, on the unseen SERV-CT dataset, their best models show over a 50% reduction in error.
- Temporal Smoothness: The proposed temporal consistency loss directly addresses and improves the temporal smoothness of depth predictions, a critical factor for real-time surgical applications.
Limitations and Future Work
- Ignoring Axial Motion: The temporal consistency loss, as formulated, ignores motion along the optical axis. The authors hypothesize that sufficient positive supervision signal can still be extracted during training, but this remains a potential area for improvement.
- Multitask Learning Impact: While explored, multitask learning for binary tool segmentation was found to dramatically reduce depth estimation performance, suggesting a need for careful integration or separate models for different tasks.
- Future Research: The paper releases their code, models, and the Meta-MED dataset, aiming to establish a strong benchmark for future research in surgical depth estimation. Further work could explore more sophisticated ways to handle axial motion or investigate other multitask learning strategies.
Summary updated