TL;DR
Metric3D v2 is a geometric foundation model that achieves state-of-the-art, zero-shot metric depth and surface normal estimation from a single image. By resolving camera-induced metric ambiguity and jointly optimizing depth and normals through a recurrent refinement module, it enables accurate 3D reconstruction and real-world metrology from arbitrary "in-the-wild" images.
What Problem Does This Paper Address?
Monocular depth estimation—predicting distance from a single viewpoint—is a fundamental computer vision challenge. However, most existing models suffer from two major limitations:
- Metric Ambiguity: Because different cameras have different focal lengths, the same object can appear at different sizes in an image. To avoid this "ambiguity," most state-of-the-art (SoTA) models learn affine-invariant depth (relative distance) rather than metric depth (actual meters). This makes them useless for tasks requiring real-world measurements, like robotics or metrology.
- Data Scarcity for Normals: While depth data is relatively abundant, high-quality surface normal labels (which describe the orientation of surfaces) are rare, especially for outdoor scenes. This limits the generalization of normal estimators.
Metric3D v2 tackles these by proposing a way to train on massive, diverse datasets while still recovering true real-world scales.
Methods
The authors introduce three key innovations to bridge the gap between relative and metric geometry:
- Canonical Camera Space Transformation (CSTM): To handle images from thousands of different camera models, the authors transform all training data into a "canonical" camera space. This involves either rescaling the ground-truth depth labels or resizing the input images based on their focal lengths. This "standardizes" the geometry, allowing the network to learn a consistent mapping from appearance to metric distance.
- Joint Depth-Normal Optimization: Inspired by optical flow architectures like RAFT, the model uses a recurrent refinement module (ConvGRU) to iteratively update both depth and surface normals. Crucially, it uses a depth-normal consistency loss, allowing the model to "distill" geometric knowledge from abundant depth labels into the normal estimator, even when explicit normal labels are missing.
- Random Proposal Normalization Loss (RPNL): Standard global normalization often "squeezes" fine-grained depth details. RPNL applies normalization to random image patches, forcing the model to preserve local geometric contrast and sharp boundaries.
Datasets
The model is a true "foundation model," trained on a massive scale:
- Training: Over 16 million images from 16 diverse datasets, including autonomous driving (Waymo, DDAD, NuScenes), indoor scenes (ScanNet, Matterport3D), and synthetic data (Hypersim, Virtual KITTI).
- Evaluation: Tested on 16 different benchmarks to prove zero-shot generalization, including NYUv2, KITTI, ETH3D, and DIODE.
Key Insights and Ideas
- Focal Length is the Key: The authors identify that focal length is the critical missing piece for metric recovery. By accounting for it during training via CSTM, they prove that a single model can generalize across wildly different camera optics (from pinhole to fisheye).
- Geometry is Complementary: Depth excels at global structure but often blurs local details; surface normals are metric-agnostic but preserve sharp local geometry. Coupling them in a recurrent loop allows each task to correct the other's errors.
- Learning Normals from Depth: By enforcing consistency, the model can learn to predict high-quality surface normals for outdoor scenes where no ground-truth normal labels exist, simply by looking at the predicted metric 3D structure.
Key Results
Metric3D v2 establishes a new state-of-the-art across nearly every benchmark it touches.
- Quantitative: It ranks 1st on 16 benchmarks for metric depth, affine-invariant depth, and surface normals. It significantly outperforms other heavyweights like Marigold and DepthAnything on standard benchmarks like NYUv2 and KITTI.
- Qualitative: The model produces significantly sharper details and fewer artifacts than previous methods, especially on complex "in-the-wild" internet images.
- Downstream Impact: When plugged into a monocular SLAM system (Droid-SLAM), it drastically reduces scale drift, enabling dense, metrically accurate mapping of large-scale outdoor scenes.
Limitations and Future Work
While highly robust, the authors note that the model is not specifically designed for video consistency, though it demonstrates promising stability in multi-view scenarios. Future work could involve explicitly incorporating temporal constraints for video-based 3D reconstruction. Additionally, while the model handles many camera types, it still relies on knowing (or estimating) the camera's focal length to perform accurate metrology in the wild.
Summary updated