Metric3Dv2: A Versatile Monocular Geometric Foundation Model for Zero-shot Metric Depth and Surface Normal Estimation

Mu Hu, Wei Yin, Chi Zhang, Zhipeng Cai, Xiaoxiao Long, Kaixuan Wang, Hao Chen, Gang Yu, Chunhua Shen, Shaojie Shen

TL;DR

Metric3D v2 is a geometric foundation model that achieves state-of-the-art, zero-shot metric depth and surface normal estimation from a single image. By resolving camera-induced metric ambiguity and jointly optimizing depth and normals through a recurrent refinement module, it enables accurate 3D reconstruction and real-world metrology from arbitrary "in-the-wild" images.

What Problem Does This Paper Address?

Monocular depth estimation—predicting distance from a single viewpoint—is a fundamental computer vision challenge. However, most existing models suffer from two major limitations:

  1. Metric Ambiguity: Because different cameras have different focal lengths, the same object can appear at different sizes in an image. To avoid this "ambiguity," most state-of-the-art (SoTA) models learn affine-invariant depth (relative distance) rather than metric depth (actual meters). This makes them useless for tasks requiring real-world measurements, like robotics or metrology.
  2. Data Scarcity for Normals: While depth data is relatively abundant, high-quality surface normal labels (which describe the orientation of surfaces) are rare, especially for outdoor scenes. This limits the generalization of normal estimators.

Metric3D v2 tackles these by proposing a way to train on massive, diverse datasets while still recovering true real-world scales.

Methods

The authors introduce three key innovations to bridge the gap between relative and metric geometry:

  • Canonical Camera Space Transformation (CSTM): To handle images from thousands of different camera models, the authors transform all training data into a "canonical" camera space. This involves either rescaling the ground-truth depth labels or resizing the input images based on their focal lengths. This "standardizes" the geometry, allowing the network to learn a consistent mapping from appearance to metric distance.
  • Joint Depth-Normal Optimization: Inspired by optical flow architectures like RAFT, the model uses a recurrent refinement module (ConvGRU) to iteratively update both depth and surface normals. Crucially, it uses a depth-normal consistency loss, allowing the model to "distill" geometric knowledge from abundant depth labels into the normal estimator, even when explicit normal labels are missing.
  • Random Proposal Normalization Loss (RPNL): Standard global normalization often "squeezes" fine-grained depth details. RPNL applies normalization to random image patches, forcing the model to preserve local geometric contrast and sharp boundaries.
Fig. 4 – Overall methodology. Our method takes a single image to predict the metric depth and surface normal simultaneously. We apply large-scale data training directly for metric depth estimation rather than affine invariant depth, enabling end-to-end zero-shot metric depth estimation for various applications using a single model. For normals, we enable learning from depth labels only, alleviating the demand for dense reconstruction to generate large-scale normal labels.

Datasets

The model is a true "foundation model," trained on a massive scale:

  • Training: Over 16 million images from 16 diverse datasets, including autonomous driving (Waymo, DDAD, NuScenes), indoor scenes (ScanNet, Matterport3D), and synthetic data (Hypersim, Virtual KITTI).
  • Evaluation: Tested on 16 different benchmarks to prove zero-shot generalization, including NYUv2, KITTI, ETH3D, and DIODE.

Key Insights and Ideas

  • Focal Length is the Key: The authors identify that focal length is the critical missing piece for metric recovery. By accounting for it during training via CSTM, they prove that a single model can generalize across wildly different camera optics (from pinhole to fisheye).
  • Geometry is Complementary: Depth excels at global structure but often blurs local details; surface normals are metric-agnostic but preserve sharp local geometry. Coupling them in a recurrent loop allows each task to correct the other's errors.
  • Learning Normals from Depth: By enforcing consistency, the model can learn to predict high-quality surface normals for outdoor scenes where no ground-truth normal labels exist, simply by looking at the predicted metric 3D structure.

Key Results

Metric3D v2 establishes a new state-of-the-art across nearly every benchmark it touches.

  • Quantitative: It ranks 1st on 16 benchmarks for metric depth, affine-invariant depth, and surface normals. It significantly outperforms other heavyweights like Marigold and DepthAnything on standard benchmarks like NYUv2 and KITTI.
  • Qualitative: The model produces significantly sharper details and fewer artifacts than previous methods, especially on complex "in-the-wild" internet images.
  • Downstream Impact: When plugged into a monocular SLAM system (Droid-SLAM), it drastically reduces scale drift, enabling dense, metrically accurate mapping of large-scale outdoor scenes.
Fig. 1 – Comparisons with SoTA methods on 16 depth and normal benchmarks. Radar-map of our Metric3D V2 v.s. SoTA methods from different works, on (1) Metric depth benchmarks, see ‘(Metric-depth)’. (2) Affine-invariant depth benchmarks, see ‘(Affine-invariant-depth)’. (3) Sur- face normal benchmarks, see ‘(Normal)’. Zero-shot testing is denoted by ‘♠’. Here δ1 percentage accuracy is used for depth benchmarks and 30◦percentage accuracy is for normal. Both higher values are for better performance. We establish new SoTA on a wide range of depth and normal benchmarks.
Fig. 2 – Surface normal (N) and monocular depth (D) comparisons on diverse web images. Our method, directly estimating metric depths and surface normals, shows powerful generalization in a variety of scenarios, including indoor, outdoor, poor- visibility, motion blurred, and fisheye images. Visualized results come from our ViT-large-backbone estimator. Marigold is a strong and robust diffusion-based monocular depth estimation method, but its recovered surface normals from the depth show various artifacts.

Limitations and Future Work

While highly robust, the authors note that the model is not specifically designed for video consistency, though it demonstrates promising stability in multi-view scenarios. Future work could involve explicitly incorporating temporal constraints for video-based 3D reconstruction. Additionally, while the model handles many camera types, it still relies on knowing (or estimating) the camera's focal length to perform accurate metrology in the wild.

Fig. 3 – Top (metrology for a complex scene): we use two phones (iPhone14 pro and Samsung Galaxy S23) to capture the scene and measure the size of several objects, including a drone which has never occurred in the whole training set. With the photos’ metadata, we perform 3D metric reconstruction and then measure object sizes (marked in red), which are close to the ground truth (marked in yellow). Compared with ZoeDepth [36], our measured sizes are closer to ground truth. Bottom (dense SLAM mapping): existing SoTA mono-SLAM methods usually face scale drift problems (see the red arrows) in large-scale scenes and are unable to achieve the metric scale, while, naively inputting our metric depth, Droid-SLAM [37] can recover much more accurate trajectory and perform the metric dense mapping (see the red measurements). Note that all testing data are unseen to our model.

Summary updated

Abstract

We introduce Metric3D v2, a geometric foundation model for zero-shot metric depth and surface normal estimation from a single image, which is crucial for metric 3D recovery. While depth and normal are geometrically related and highly complimentary, they present distinct challenges. SoTA monocular depth methods achieve zero-shot generalization by learning affine-invariant depths, which cannot recover real-world metrics. Meanwhile, SoTA normal estimation methods have limited zero-shot performance due to the lack of large-scale labeled data. To tackle these issues, we propose solutions for both metric depth estimation and surface normal estimation. For metric depth estimation, we show that the key to a zero-shot single-view model lies in resolving the metric ambiguity from various camera models and large-scale data training. We propose a canonical camera space transformation module, which explicitly addresses the ambiguity problem and can be effortlessly plugged into existing monocular models. For surface normal estimation, we propose a joint depth-normal optimization module to distill diverse data knowledge from metric depth, enabling normal estimators to learn beyond normal labels. Equipped with these modules, our depth-normal models can be stably trained with over 16 million of images from thousands of camera models with different-type annotations, resulting in zero-shot generalization to in-the-wild images with unseen camera settings. Our method enables the accurate recovery of metric 3D structures on randomly collected internet images, paving the way for plausible single-image metrology. Our project page is at https://JUGGHM.github.io/Metric3Dv2.
BibTeX
@article{hu2024metric,
  author = {Mu Hu and Wei Yin and Chi Zhang and Zhipeng Cai and Xiaoxiao Long and Kaixuan Wang and Hao Chen and Gang Yu and Chunhua Shen and Shaojie Shen},
  title = {{Metric3Dv2: A Versatile Monocular Geometric Foundation Model for Zero-shot Metric Depth and Surface Normal Estimation}},
  journal = {IEEE Trans. Pattern Anal. Mach. Intell.},
  year = {2024},
  doi = {10.1109/tpami.2024.3444912},
}
Figure