TL;DR
LangDepth (SemanticBins) is a novel monocular depth estimation architecture that uses a CLIP-based semantic prior to define a scene's metric depth range before dense visual decoding begins. By explicitly separating scene-level scale (e.g., "this is a highway" vs. "this is a kitchen") from local geometry, the model achieves state-of-the-art cross-domain performance without requiring massive web-scale training data.
What Problem Does This Paper Address?
Predicting the absolute "metric" depth of a scene from a single 2D image is notoriously difficult because of scale ambiguity. A small room and a vast outdoor landscape look fundamentally different, yet most depth decoders try to infer this global scale implicitly from local visual features.
While recent "web-scale" models solve this by training on tens of millions of images, this is computationally expensive and often results in "black box" behavior. The authors of LangDepth ask a simpler question: Can we use high-level semantic knowledge—the kind encoded in vision-language models like CLIP—to set a "range anchor" for the scene? If the model knows it's looking at a corridor, it can constrain its predictions to a 0–10 meter range before it even looks at the individual pixels.
Methods
LangDepth employs a dual-branch architecture that meets at a final fusion step:
- The Semantic Branch: This branch uses a frozen CLIP image encoder. Instead of requiring text captions, it routes image features through a learnable Concept Bank (65 keys representing scene types and geometric properties). This produces a compact scene descriptor $z$.
- The Scene Prior Module (SPM): An MLP that takes the descriptor $z$ and predicts a scene-conditioned metric interval $(d_{min}, d_{max})$. This defines the "bins" or the search space for depth.
- The Vision Branch: A heavy-duty DINOv2-L backbone with a Feature Pyramid Network (FPN) extracts dense visual features.
- Per-Pixel Assignment Network (PPAN): This network assigns each pixel to the depth bins defined by the SPM. Crucially, it uses FiLM modulation to condition these visual features on the semantic descriptor $z$.
- Confidence-Gated Fusion: The final depth map is a blend of the dense visual estimate and the global SPM prior, controlled by a learned "gate" that decides how much to trust local visual evidence versus the global scene context.
Datasets
The model was trained jointly on two primary datasets to cover both indoor and outdoor scales:
- NYU Depth V2: Indoor scenes with a 10m depth cap.
- KITTI Eigen: Outdoor driving scenes with an 80m depth cap.
To test generalization (zero-shot transfer), the authors evaluated the model on:
- DIODE indoor
- iBims-1
- SUN RGB-D
Key Insights and Ideas
- Semantics Set the Stage: The SPM successfully learns to separate scale regimes. It predicts narrow ranges (~8–10m) for indoor datasets and much wider ranges (mean ~55.6m) for KITTI, all without being told which dataset an image belongs to.
- The Gate is Domain-Adaptive: The confidence gate reveals an interesting behavior: on outdoor KITTI scenes, the model is "SPM-dominant" (relying 90% on the global prior). Indoors, the model relies more on the dense visual branch. This suggests that global context is a more reliable cue for outdoor scale than local pixel variation.
- Structural Importance of Context: By perturbing the scene descriptor (e.g., zeroing it out or providing "wrong domain" context), the authors proved that the semantic branch is "structurally active." Corrupting the context leads to massive accuracy drops, confirming the model isn't just using CLIP as a minor auxiliary feature.
Key Results
LangDepth achieves impressive results using a relatively modest supervised budget:
- Accuracy: Reaches a $\delta_1$ (threshold accuracy) of 0.960 on both NYU and KITTI.
- Transfer Learning: Outperforms other non-web-scale methods on held-out benchmarks, achieving 0.887 on DIODE indoor and 0.946 on iBims-1.
- Efficiency: Despite the dual-branch approach, it runs at 13.8 FPS on an H100 GPU, making it practical for real-world applications.
Limitations and Future Work
- Heterogeneous Scenes: The model still struggles with SUN RGB-D, a highly diverse indoor benchmark where semantic cues can be noisier.
- Precision Trade-offs: While the model is a great generalist, it trades off some fine-scale metric precision on KITTI compared to models trained exclusively on outdoor driving data.
- Future Directions: The authors suggest improving the robustness of scene priors for complex, mixed indoor-outdoor environments as a key next step.
Summary updated