TL;DR
RAD (Retrieval-Augmented Depth) introduces a novel framework that significantly improves monocular metric depth estimation for rare and underrepresented object classes. By retrieving semantically similar RGB-D images from a reference database to serve as structural geometric proxies, RAD mimics the benefits of multi-view stereo without requiring multiple viewpoints of the exact same scene.
What Problem Does This Paper Address?
Monocular Metric Depth Estimation (MMDE)—predicting accurate, real-world distance from a single image—is a foundational pillar for autonomous driving, robotics, and augmented reality. While deep learning models have become remarkably good at this task, they suffer from a classic machine learning trap: the long-tail distribution.
When a model encounters an underrepresented or rare object class (such as a stroller on a street or a candle on a dining table), its internal geometric priors fail, leading to massive depth errors. Traditional multi-view stereo techniques solve this by matching pixels across different viewpoints, but they require multiple images of the same scene. RAD addresses this limitation by asking a clever question: Can we borrow the geometric structure of similar objects from entirely different scenes to fix single-image depth errors?
Methods
RAD adapts the popular Retrieval-Augmented Generation (RAG) paradigm from natural language processing and applies it to 3D computer vision. The pipeline consists of three core stages:
- Uncertainty-Aware Retrieval: Standard image retrieval algorithms look at global scene layouts (e.g., matching a kitchen with another kitchen). RAD instead calculates pixel-wise depth uncertainty by feeding noisy perturbations of the input image into a baseline model (DepthAnything v2). High-uncertainty regions are isolated using Segment Anything 2 (SAM2) to create a masked query. A DINOv2-based search then finds semantically similar RGB-D context samples from a reference pool (the training set).
- Spatial Correspondence: To align the input image with the retrieved context, RAD uses an off-the-shelf point-matching method (LightGlue). During training, the authors also introduce a 3D Augmentation strategy—back-projecting the input image into 3D and rendering it from random camera poses—to prevent the model from overfitting to fixed retrieved samples.
- Dual-Stream Network & Matched Cross-Attention: The input image and the retrieved RGB-D context are processed via parallel Vision Transformer (ViT) encoders. To merge this information safely, RAD introduces Matched Cross-Attention. Instead of allowing global attention (which introduces noise), it restricts attention to localized spatial neighborhoods around the matched points, ensuring that geometric data is only transferred where correspondences are highly reliable.
Datasets
The authors evaluate RAD on three widely used depth estimation benchmarks:
- NYU Depth v2 (Indoor scenes)
- KITTI (Outdoor driving scenes, Eigen split)
- Cityscapes (Urban street scenes)
Because standard benchmarks evaluate performance across all pixels collectively—which dilutes the errors made on rare objects—the authors introduce a new benchmark subset specifically targeting underrepresented classes. They define these as classes appearing in fewer than 10% of training images but occurring more than five times (resulting in 231 rare classes for NYU and 22 for KITTI).
Key Insights and Ideas
- Uncertainty as a Rarity Signal: The authors' core "aha moment" is that a depth network's internal uncertainty is an excellent proxy for object rarity. Leveraging this signal allows the model to ignore the background layout and focus purely on retrieving context for the problematic object.
- Semantic Multi-View Stereo: Traditional multi-view geometry relies on rigid, overlapping camera views. RAD proves that "semantic multi-view"—using a completely different environment that contains a similar object—can provide a strong enough geometric proxy to correct monocular depth.
- Locality is Vital for Fusion: Simply applying standard cross-attention between the input and retrieved streams degrades performance. Restricting the attention mechanism to local neighborhoods around matched points is essential to filter out incorrect correspondences.
Key Results
RAD (built on a DepthAnything v2 backbone) achieves state-of-the-art performance on underrepresented classes while maintaining highly competitive results on standard "all-classes" benchmarks:
- NYU Depth v2: RAD-Large reduced the relative absolute error (AbsRel) on rare classes by 29.2% and root mean squared error (RMS) by 11.9% compared to the baseline DepthAnything v2.
- KITTI: Achieved a 13.3% relative improvement in AbsRel and a 39.5% improvement in $\text{RMS}_{\text{log}}$ on underrepresented classes.
- Cityscapes: Outperformed the best non-RAD baseline (Metric3D v2) by 7.2% in AbsRel and 28.0% in RMS.
Qualitatively, RAD successfully resolves depth artifacts on highly uncertain objects like lamps, strollers, and trains, accurately carving out their geometry where baseline models produce blurred or flattened depth maps.
Limitations and Future Work
Detailed limitations are deferred to the supplementary material of the paper. However, the main text highlights a few practical boundaries:
- Diminishing Returns in Retrieval: Ablation studies show that performance peaks when retrieving exactly four context images. Adding a fifth image introduces irrelevant or misaligned content, which slightly degrades performance.
- Reference Pool Dependency: The framework inherently relies on the reference pool containing a clean, diverse set of RGB-D examples for the rare classes it encounters.
- Future Directions: Promising avenues include optimizing the inference-time retrieval overhead and extending this retrieval-augmented framework to other dense 3D prediction tasks like surface normal estimation or 3D semantic occupancy.
Summary updated