Learning to Adapt CLIP for Few-Shot Monocular Depth Estimation

Xueting Hu, Ce Zhang, Yi Zhang, Bowen Hai, Ke Yu, Zhihai He

Abstract

Pre-trained Vision-Language Models (VLMs), such as CLIP, have shown enhanced performance across a range of tasks that involve the integration of visual and linguistic modalities. When CLIP is used for depth estimation tasks, the patches, divided from the input images, can be combined with a series of semantic descriptions of the depth information to obtain similarity results. The coarse estimation of depth is then achieved by weighting and summing the depth values, called depth bins, corresponding to the predefined semantic descriptions. The zero-shot approach circumvents the computational and time-intensive nature of traditional fully-supervised depth estimation methods. However, this method, utilizing fixed depth bins, may not effectively generalize as images from different scenes may exhibit distinct depth distributions. To address this challenge, we propose a few-shot-based method which learns to adapt the VLMs for monocular depth estimation to balance training costs and generalization capabilities. Specifically, it assigns different depth bins for different scenes, which can be selected by the model during inference. Additionally, we incorporate learnable prompts to preprocess the input text to convert the easily human-understood text into easily model-understood vectors and further enhance the performance. With only one image per scene for training, our extensive experiment results on the NYU V2 and KITTI dataset demonstrate that our method outperforms the previous state-of-the-art method by up to 10.6% in terms of MARE.

The Challenge of Scene-Dependent Depth

Monocular Depth Estimation (MDE)—predicting 3D distance from a single 2D image—is a cornerstone of autonomous driving and robotics. Traditionally, this requires massive datasets of "ground truth" laser scans, which are expensive and time-consuming to collect.

Recently, researchers have explored using CLIP (Contrastive Language-Image Pre-Training) for this task. CLIP possesses a semantic understanding of spatial relationships (e.g., "far" vs. "close") from its training on millions of internet images. However, existing zero-shot methods like DepthCLIP rely on fixed "depth bins"—predefined numerical values for labels like "giant" or "unseen."

The primary limitation of this approach is that depth is relative. A "far" object in a small bathroom is physically much closer than a "far" object in a large bedroom or on an open highway. As shown in the figure below, different scenes exhibit vastly different depth distributions. Using the same fixed depth values for every environment leads to significant generalization errors.

Overview of the proposed approach compared to DepthCLIP, highlighting scene-dependent depth distributions and learnable components.

Methodology: Learnable Prompts and Scene-Specific Codebooks

To address these limitations, this research proposes a few-shot adaptation strategy that moves away from "one-size-fits-all" depth values. The architecture introduces three core innovations:

  1. Learnable Prompts: Instead of using human-written sentences like "This object is [depth class]," the model learns optimized mathematical vectors ($V_1, V_2, \dots, V_n$). These "soft prompts" translate human concepts into a high-dimensional language that the CLIP text encoder understands more precisely for depth estimation tasks.
  2. Scene Encoder: The model processes the entire image to identify the scene type (e.g., "cafe" vs. "office"). This global context is crucial for setting the appropriate scale for the environment.
  3. Learnable Depth Codebook: Instead of a single list of depth values, the model maintains a codebook. Once the Scene Encoder identifies the environment, the model selects a specific set of depth bins tailored to that scene. This allows the model to "re-calibrate" its internal ruler based on the context.
  4. Few-Shot Efficiency: The model is trained using only one image per scene category. During training, only the prompt vectors and the depth codebook (roughly 8,000 parameters) are updated, while the massive CLIP backbone remains frozen. This prevents "catastrophic forgetting" and enables extremely fast training—approximately 70 minutes on a single consumer GPU.

Experimental Results

The researchers evaluated their approach on two industry-standard benchmarks: NYU V2 (indoor scenes) and KITTI (outdoor driving scenes).

Key findings include:

  • Accuracy Boost: On the NYU V2 dataset, the method improved the Mean Absolute Relative Error (MARE) by 10.6% compared to the previous state-of-the-art zero-shot method, DepthCLIP.
  • Competitive Performance: Remarkably, this few-shot approach (using only 27 images for training) outperformed some older fully-supervised methods that were trained on thousands of images.
  • Qualitative Sharpness: Visual results indicate that learnable prompts help the model better distinguish object boundaries and produce smoother depth transitions compared to rigid, manually-defined prompts.

Conclusion and Future Work

By mapping CLIP’s semantic understanding of distance to flexible, scene-specific metric units, this method achieves a new balance between training cost and generalization. While the system currently relies on predefined scene categories, future work may explore handling dynamic environments where the scale of the scene changes as the camera moves, or upgrading the underlying CLIP backbone to larger Vision Transformers (ViT) to further push the performance ceiling.

Summary updated

Abstract

Pre-trained Vision-Language Models (VLMs), such as CLIP, have shown enhanced performance across a range of tasks that involve the integration of visual and linguistic modalities. When CLIP is used for depth estimation tasks, the patches, divided from the input images, can be combined with a series of semantic descriptions of the depth information to obtain similarity results. The coarse estimation of depth is then achieved by weighting and summing the depth values, called depth bins, corresponding to the predefined semantic descriptions. The zero-shot approach circumvents the computational and time-intensive nature of traditional fully-supervised depth estimation methods. However, this method, utilizing fixed depth bins, may not effectively generalize as images from different scenes may exhibit distinct depth distributions. To address this challenge, we propose a few-shot-based method which learns to adapt the VLMs for monocular depth estimation to balance training costs and generalization capabilities. Specifically, it assigns different depth bins for different scenes, which can be selected by the model during inference. Additionally, we incorporate learnable prompts to preprocess the input text to convert the easily human-understood text into easily model-understood vectors and further enhance the performance. With only one image per scene for training, our extensive experiment results on the NYU V2 and KITTI dataset demonstrate that our method outperforms the previous state-of-the-art method by up to 10.6\% in terms of MARE.
BibTeX
@inproceedings{hu2024learning,
  author = {Xueting Hu and Ce Zhang and Yi Zhang and Bowen Hai and Ke Yu and Zhihai He},
  title = {{Learning to Adapt CLIP for Few-Shot Monocular Depth Estimation}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2024},
  doi = {10.1109/wacv57701.2024.00550},
}
Figure