Learning to Adapt CLIP for Few-Shot Monocular Depth Estimation

Xueting Hu, Ce Zhang, Yi Zhang, Bowen Hai, Ke Yu, Zhihai He

Abstract

Monocular depth estimation (MDE) typically requires extensive ground-truth data and intensive training. While zero-shot methods leveraging vision-language models like CLIP have emerged, they often struggle with scene-specific scale variations. This paper introduces a few-shot adaptation strategy for CLIP that utilizes learnable text prompts and a scene-specific "depth codebook." By optimizing only 8,000 parameters with as little as one image per scene category, the proposed method outperforms previous zero-shot approaches by over 10% on standard benchmarks like NYU V2 and KITTI, bridging the gap between zero-shot and fully supervised models.

Introduction

Predicting 3D distance from a single 2D image—monocular depth estimation—is critical for robotics and autonomous systems. Traditional supervised approaches rely on massive datasets of depth maps. Recent "zero-shot" attempts, such as DepthCLIP, use pre-trained vision-language models to match image patches with descriptive text (e.g., "this object is far"). However, these methods use fixed depth bins that fail to account for environmental context; "far" in a bathroom is vastly different from "far" in an outdoor highway.

Methodology: Learnable Prompts and Depth Codebooks

The core innovation of this work is the transition from hard-coded logic to learnable components that adapt to specific environments.

Comparison between fixed zero-shot baselines and the proposed learnable prompt and depth codebook approach.

Learnable Prompts

Instead of relying on static human-written templates like "This object is [CLASS]," the model learns optimized vectors. These learnable prompts translate abstract depth concepts into a high-dimensional space that the CLIP text encoder can more effectively align with visual features for the specific geometry of depth sensing.

Scene-Specific Depth Codebook

To solve the scale ambiguity problem, the authors introduce a "depth codebook." Rather than using a single set of depth values for all images, the model maintains optimized depth bins tailored to specific scene categories (e.g., "office," "kitchen," "study"). During inference, a CLIP-based scene encoder identifies the environment type, and the model selects the corresponding bins from the codebook to weight the final depth calculation.

Efficiency in Training

The model is remarkably parameter-efficient. While traditional models might use over 50 million parameters, this method updates only 8,000. Training requires only a handful of images (e.g., 27 images for the NYU V2 dataset) and can be completed in approximately 70 minutes on a single consumer GPU.

Experimental Results

The approach was evaluated on the NYU V2 (indoor) and KITTI (outdoor) datasets.

Qualitative example of monocular depth estimation showing the input image, predicted depth map, and the probability distribution over depth bins.

Performance Gains

On the NYU V2 dataset, the method achieved a 10.6% improvement in Mean Absolute Relative Error (MARE) over the previous state-of-the-art zero-shot model, DepthCLIP. Qualitatively, the resulting depth maps are sharper and more accurately scaled, avoiding the "flat" predictions common in non-adaptive models.

Comparison with Supervised Models

Despite the minimal data requirements, the few-shot adaptation outperforms some older fully-supervised models like Make3D. This suggests that the pre-trained knowledge in CLIP, when properly adapted, is a powerful prior for geometric tasks.

Discussion and Future Work

The success of this method highlights that context is essential for depth estimation. By making depth bins scene-dependent, the model dynamically adapts its scale. However, some limitations remain. The current patch-based approach can lead to "blocky" depth maps, as every pixel in a patch is assigned the same value. Additionally, the system's accuracy is tied to the reliability of the initial scene classification.

Future research will focus on moving toward pixel-level estimation for smoother depth maps and exploring more sophisticated prompt-tuning techniques to further narrow the performance gap with large-scale supervised models.

Summary updated

Abstract

Pre-trained Vision-Language Models (VLMs), such as CLIP, have shown enhanced performance across a range of tasks that involve the integration of visual and linguistic modalities. When CLIP is used for depth estimation tasks, the patches, divided from the input images, can be combined with a series of semantic descriptions of the depth information to obtain similarity results. The coarse estimation of depth is then achieved by weighting and summing the depth values, called depth bins, corresponding to the predefined semantic descriptions. The zero-shot approach circumvents the computational and time-intensive nature of traditional fully-supervised depth estimation methods. However, this method, utilizing fixed depth bins, may not effectively generalize as images from different scenes may exhibit distinct depth distributions. To address this challenge, we propose a few-shot-based method which learns to adapt the VLMs for monocular depth estimation to balance training costs and generalization capabilities. Specifically, it assigns different depth bins for different scenes, which can be selected by the model during inference. Additionally, we incorporate learnable prompts to preprocess the input text to convert the easily human-understood text into easily model-understood vectors and further enhance the performance. With only one image per scene for training, our extensive experiment results on the NYU V2 and KITTI dataset demonstrate that our method outperforms the previous state-of-the-art method by up to 10.6% in terms of MARE 1 .
BibTeX
@inproceedings{hu2024learning,
  author = {Xueting Hu and Ce Zhang and Yi Zhang and Bowen Hai and Ke Yu and Zhihai He},
  title = {{Learning to Adapt CLIP for Few-Shot Monocular Depth Estimation}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2024},
  doi = {10.1109/wacv57701.2024.00550},
}
Figure