RAD: Retrieval-Augmented Monocular Metric Depth Estimation for Underrepresented Classes

Michael Baltaxe, Dan Levi, Sagie Benaim

TL;DR

RAD (Retrieval-Augmented Depth) introduces a novel framework that significantly improves monocular metric depth estimation for rare and underrepresented object classes. By retrieving semantically similar RGB-D images from a reference database to serve as structural geometric proxies, RAD mimics the benefits of multi-view stereo without requiring multiple viewpoints of the exact same scene.

What Problem Does This Paper Address?

Monocular Metric Depth Estimation (MMDE)—predicting accurate, real-world distance from a single image—is a foundational pillar for autonomous driving, robotics, and augmented reality. While deep learning models have become remarkably good at this task, they suffer from a classic machine learning trap: the long-tail distribution.

When a model encounters an underrepresented or rare object class (such as a stroller on a street or a candle on a dining table), its internal geometric priors fail, leading to massive depth errors. Traditional multi-view stereo techniques solve this by matching pixels across different viewpoints, but they require multiple images of the same scene. RAD addresses this limitation by asking a clever question: Can we borrow the geometric structure of similar objects from entirely different scenes to fix single-image depth errors?

RAD Overview

Methods

RAD adapts the popular Retrieval-Augmented Generation (RAG) paradigm from natural language processing and applies it to 3D computer vision. The pipeline consists of three core stages:

  1. Uncertainty-Aware Retrieval: Standard image retrieval algorithms look at global scene layouts (e.g., matching a kitchen with another kitchen). RAD instead calculates pixel-wise depth uncertainty by feeding noisy perturbations of the input image into a baseline model (DepthAnything v2). High-uncertainty regions are isolated using Segment Anything 2 (SAM2) to create a masked query. A DINOv2-based search then finds semantically similar RGB-D context samples from a reference pool (the training set).
  2. Spatial Correspondence: To align the input image with the retrieved context, RAD uses an off-the-shelf point-matching method (LightGlue). During training, the authors also introduce a 3D Augmentation strategy—back-projecting the input image into 3D and rendering it from random camera poses—to prevent the model from overfitting to fixed retrieved samples.
  3. Dual-Stream Network & Matched Cross-Attention: The input image and the retrieved RGB-D context are processed via parallel Vision Transformer (ViT) encoders. To merge this information safely, RAD introduces Matched Cross-Attention. Instead of allowing global attention (which introduces noise), it restricts attention to localized spatial neighborhoods around the matched points, ensuring that geometric data is only transferred where correspondences are highly reliable.
RAD Pipeline

Datasets

The authors evaluate RAD on three widely used depth estimation benchmarks:

  • NYU Depth v2 (Indoor scenes)
  • KITTI (Outdoor driving scenes, Eigen split)
  • Cityscapes (Urban street scenes)

Because standard benchmarks evaluate performance across all pixels collectively—which dilutes the errors made on rare objects—the authors introduce a new benchmark subset specifically targeting underrepresented classes. They define these as classes appearing in fewer than 10% of training images but occurring more than five times (resulting in 231 rare classes for NYU and 22 for KITTI).

Key Insights and Ideas

  • Uncertainty as a Rarity Signal: The authors' core "aha moment" is that a depth network's internal uncertainty is an excellent proxy for object rarity. Leveraging this signal allows the model to ignore the background layout and focus purely on retrieving context for the problematic object.
  • Semantic Multi-View Stereo: Traditional multi-view geometry relies on rigid, overlapping camera views. RAD proves that "semantic multi-view"—using a completely different environment that contains a similar object—can provide a strong enough geometric proxy to correct monocular depth.
  • Locality is Vital for Fusion: Simply applying standard cross-attention between the input and retrieved streams degrades performance. Restricting the attention mechanism to local neighborhoods around matched points is essential to filter out incorrect correspondences.

Key Results

RAD (built on a DepthAnything v2 backbone) achieves state-of-the-art performance on underrepresented classes while maintaining highly competitive results on standard "all-classes" benchmarks:

  • NYU Depth v2: RAD-Large reduced the relative absolute error (AbsRel) on rare classes by 29.2% and root mean squared error (RMS) by 11.9% compared to the baseline DepthAnything v2.
  • KITTI: Achieved a 13.3% relative improvement in AbsRel and a 39.5% improvement in $\text{RMS}_{\text{log}}$ on underrepresented classes.
  • Cityscapes: Outperformed the best non-RAD baseline (Metric3D v2) by 7.2% in AbsRel and 28.0% in RMS.

Qualitatively, RAD successfully resolves depth artifacts on highly uncertain objects like lamps, strollers, and trains, accurately carving out their geometry where baseline models produce blurred or flattened depth maps.

Qualitative Results

Limitations and Future Work

Detailed limitations are deferred to the supplementary material of the paper. However, the main text highlights a few practical boundaries:

  • Diminishing Returns in Retrieval: Ablation studies show that performance peaks when retrieving exactly four context images. Adding a fifth image introduces irrelevant or misaligned content, which slightly degrades performance.
  • Reference Pool Dependency: The framework inherently relies on the reference pool containing a clean, diverse set of RGB-D examples for the rare classes it encounters.
  • Future Directions: Promising avenues include optimizing the inference-time retrieval overhead and extending this retrieval-augmented framework to other dense 3D prediction tasks like surface normal estimation or 3D semantic occupancy.

Summary updated

Abstract

Monocular Metric Depth Estimation (MMDE) is essential for physically intelligent systems, yet accurate depth estimation for underrepresented classes in complex scenes remains a persistent challenge. To address this, we propose RAD, a retrieval-augmented framework that approximates the benefits of multi-view stereo by utilizing retrieved neighbors as structural geometric proxies. Our method first employs an uncertainty-aware retrieval mechanism to identify low-confidence regions in the input and retrieve RGB-D context samples containing semantically similar content. We then process both the input and retrieved context via a dual-stream network and fuse them using a matched cross-attention module, which transfers geometric information only at reliable point correspondences. Evaluations on NYU Depth v2, KITTI, and Cityscapes demonstrate that RAD significantly outperforms state-of-the-art baselines on underrepresented classes, reducing relative absolute error by 29.2% on NYU Depth v2, 13.3% on KITTI, and 7.2% on Cityscapes, while maintaining competitive performance on standard in-domain benchmarks. Project page: https://michaelbaltaxe.github.io/rad
BibTeX
@inproceedings{baltaxe2026rad,
  author = {Michael Baltaxe and Dan Levi and Sagie Benaim},
  title = {{RAD: Retrieval-Augmented Monocular Metric Depth Estimation for Underrepresented Classes}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2026},
  url = {https://openreview.net/forum?id=bZfWw7cgcg},
}
Figure