Abstract
Monocular depth estimation (MDE) typically requires extensive ground-truth data and intensive training. While zero-shot methods leveraging vision-language models like CLIP have emerged, they often struggle with scene-specific scale variations. This paper introduces a few-shot adaptation strategy for CLIP that utilizes learnable text prompts and a scene-specific "depth codebook." By optimizing only 8,000 parameters with as little as one image per scene category, the proposed method outperforms previous zero-shot approaches by over 10% on standard benchmarks like NYU V2 and KITTI, bridging the gap between zero-shot and fully supervised models.
Introduction
Predicting 3D distance from a single 2D image—monocular depth estimation—is critical for robotics and autonomous systems. Traditional supervised approaches rely on massive datasets of depth maps. Recent "zero-shot" attempts, such as DepthCLIP, use pre-trained vision-language models to match image patches with descriptive text (e.g., "this object is far"). However, these methods use fixed depth bins that fail to account for environmental context; "far" in a bathroom is vastly different from "far" in an outdoor highway.
Methodology: Learnable Prompts and Depth Codebooks
The core innovation of this work is the transition from hard-coded logic to learnable components that adapt to specific environments.
Learnable Prompts
Instead of relying on static human-written templates like "This object is [CLASS]," the model learns optimized vectors. These learnable prompts translate abstract depth concepts into a high-dimensional space that the CLIP text encoder can more effectively align with visual features for the specific geometry of depth sensing.
Scene-Specific Depth Codebook
To solve the scale ambiguity problem, the authors introduce a "depth codebook." Rather than using a single set of depth values for all images, the model maintains optimized depth bins tailored to specific scene categories (e.g., "office," "kitchen," "study"). During inference, a CLIP-based scene encoder identifies the environment type, and the model selects the corresponding bins from the codebook to weight the final depth calculation.
Efficiency in Training
The model is remarkably parameter-efficient. While traditional models might use over 50 million parameters, this method updates only 8,000. Training requires only a handful of images (e.g., 27 images for the NYU V2 dataset) and can be completed in approximately 70 minutes on a single consumer GPU.
Experimental Results
The approach was evaluated on the NYU V2 (indoor) and KITTI (outdoor) datasets.
Performance Gains
On the NYU V2 dataset, the method achieved a 10.6% improvement in Mean Absolute Relative Error (MARE) over the previous state-of-the-art zero-shot model, DepthCLIP. Qualitatively, the resulting depth maps are sharper and more accurately scaled, avoiding the "flat" predictions common in non-adaptive models.
Comparison with Supervised Models
Despite the minimal data requirements, the few-shot adaptation outperforms some older fully-supervised models like Make3D. This suggests that the pre-trained knowledge in CLIP, when properly adapted, is a powerful prior for geometric tasks.
Discussion and Future Work
The success of this method highlights that context is essential for depth estimation. By making depth bins scene-dependent, the model dynamically adapts its scale. However, some limitations remain. The current patch-based approach can lead to "blocky" depth maps, as every pixel in a patch is assigned the same value. Additionally, the system's accuracy is tied to the reliability of the initial scene classification.
Future research will focus on moving toward pixel-level estimation for smoother depth maps and exploring more sophisticated prompt-tuning techniques to further narrow the performance gap with large-scale supervised models.
Summary updated