TL;DR
This paper tackles the challenge of zero-shot depth completion by adapting powerful monocular depth foundation models. They propose a novel "test-time visual prompt tuning" method that efficiently adjusts depth scale using sparse sensor data without modifying the foundation model's parameters, achieving state-of-the-art results with significantly reduced computational cost compared to fine-tuning.
What Problem Does This Paper Address?
Depth completion, the task of inferring dense depth maps from sparse sensor measurements (like LiDAR or depth cameras) and corresponding RGB images, is crucial for 3D reconstruction, autonomous driving, and robotics. While deep learning has advanced this field, models often struggle with generalization to unseen environments due to domain specificity and sensor variations. A promising direction is to leverage large-scale monocular depth foundation models, which can estimate initial depth from single images. However, directly integrating these models into depth completion frameworks faces two main hurdles: 1) they often require additional networks to handle sparse depth inputs, increasing complexity, and 2) adapting them to the specific scale of sparse sensor data can be computationally expensive and prone to introducing biases. This paper aims to find an efficient and effective way to adapt these foundation models for depth completion without these drawbacks.
Methods
The authors explore two main strategies for adapting monocular depth foundation models for depth completion:
- Test-Time Fine-tuning (Ours-FT): This approach involves fine-tuning the parameters of a pre-trained monocular depth foundation model during inference using only the sparse depth measurements as supervision. This allows the model to adjust its depth scale locally to match the sensor data. While effective, this method incurs significant computational costs and risks "catastrophic forgetting" of the model's original knowledge.
- Test-Time Visual Prompt Tuning (Ours-VP): This is the paper's core contribution. Instead of fine-tuning the entire foundation model, they introduce a small set of learnable parameters that form a "pixelwise visual prompt." This prompt is added directly to the input RGB image. The foundation model's parameters remain frozen, and only the prompt parameters are optimized at test time using the sparse depth as supervision. This method is designed to be parameter-efficient, adapt depth scale accurately, and avoid forgetting foundational knowledge. The pixelwise nature of the prompt allows for fine-grained, localized adjustments.
Datasets
The paper evaluates their methods on several real-world datasets:
- IBims-1: An indoor RGB-D dataset.
- VOID: Contains RGB-D frames from various sequences.
- NYUv2: A widely used indoor RGB-D dataset.
- KITTI: A dataset commonly used for autonomous driving tasks, often with LiDAR data.
- DDAD: An outdoor RGB-D driving dataset.
For evaluation, they use standard splits and generate sparse depth by randomly sampling points from the ground truth depth maps, mimicking real-world sensor data.
Key Insights and Ideas
- Test-time adaptation is powerful: The paper demonstrates that adapting foundation models at inference time, even with limited sparse data, can yield significant improvements in depth completion.
- Fine-tuning is effective but costly: Simple fine-tuning of foundation models with sparse depth supervision works surprisingly well, but the computational overhead and risk of forgetting are major limitations.
- Visual prompts offer a better trade-off: Pixelwise visual prompt tuning provides a way to achieve fine-tuning-level performance with drastically fewer learnable parameters and without modifying the foundation model itself.
- Pixelwise adaptation is key for depth: Unlike previous visual prompting methods that might operate globally or on patches, a pixelwise approach is crucial for the fine-grained adjustments needed for accurate depth scale alignment.
- Foundation models are versatile: The visual prompting approach works effectively across different foundation models, highlighting its model-agnostic nature.
Key Results
- State-of-the-art performance: The proposed visual prompt tuning (Ours-VP) achieves superior zero-shot depth completion performance across multiple benchmark datasets, often outperforming existing methods.
- Efficiency gains: Ours-VP requires significantly less memory (3991 MB) and achieves faster inference speeds (2.93s) compared to methods like Zero-DC (20.56s) and Marigold-DC (121.04s).
- Robustness to sparsity: The method maintains high accuracy even with a reduced number of sparse depth points, demonstrating robustness to varying input sparsity.
- Effective scale alignment: Visual prompting effectively corrects scale mismatches, a common issue with monocular depth estimation, by adaptively adjusting predictions at the pixel level.
- Fine-tuning variant also competitive: Even the simpler fine-tuning approach (Ours-FT) shows strong performance and is computationally efficient compared to many other methods.
Limitations and Future Work
- Computational cost of fine-tuning: While the paper introduces a more efficient method, the initial fine-tuning approach still has high computational costs and risks knowledge forgetting.
- Dataset bias: The authors mention that fine-tuning might introduce dataset-specific biases, which they mitigate by resetting parameters between evaluations.
- Future work: The paper suggests that their visual prompt tuning strategy can be applied to various foundation models, implying potential for further exploration with newer or different architectures. The adaptive nature of the prompt tuning also suggests potential for dynamic adaptation based on available computational resources.
Summary updated