Abstract
Monocular depth estimation (MDE) has traditionally faced a significant trade-off between metric accuracy and zero-shot generalization. While metric models provide absolute distance measurements, they often fail when applied to cameras with different intrinsics. Conversely, affine-invariant models generalize well across diverse scenes but lack the ability to recover true physical scale. This paper presents Metric3D, a framework that overcomes the scale-ambiguity problem by introducing a canonical camera transformation. By training on a massive dataset of 8 million images using a novel Random Proposal Normalization Loss, Metric3D achieves state-of-the-art performance in zero-shot metric depth estimation, enabling accurate 3D reconstruction from arbitrary single images.
Introduction
Estimating 3D geometry from a single 2D image is a fundamental challenge in computer vision. Applications ranging from autonomous driving to augmented reality require not just a relative understanding of which objects are closer than others, but precise "metric" measurements (e.g., distance in meters). Historically, the field has been split into two paradigms. Metric depth models are trained on specific datasets (like KITTI or NYUv2) and provide absolute scale, but they are notoriously brittle; a model trained on a high-mounted automotive camera will produce incorrect distances if applied to a handheld smartphone image. On the other hand, affine-invariant models like MiDaS prioritize generalization, working across any image from the internet, but they only provide relative depth maps that lack physical scale.
Metric3D bridges this gap, providing a model that is both metrically accurate and capable of zero-shot generalization to unseen camera systems.
The Challenge of Metric Ambiguity
The primary obstacle to universal metric depth estimation is metric ambiguity. In a pinhole camera model, the projected size of an object depends on both its physical distance and the camera's focal length. Consequently, two different cameras at different distances can produce identical images of the same object.
During large-scale training on diverse datasets, this ambiguity leads to "conflicting" labels. A neural network seeing similar visual features associated with different ground-truth distances becomes difficult to converge and fails to learn a consistent mapping. The authors identify that while sensor size and pixel dimensions are secondary, the focal length is the critical factor causing this confusion.
The Metric3D Framework
Canonical Camera Space Transformation (CSTM)
To resolve metric ambiguity, the researchers proposed the Canonical Camera Space Transformation (CSTM). The core idea is to transform all training data into a unified "virtual" camera space with a standardized focal length. This ensures that the relationship between visual appearance and depth is consistent across the entire 8-million-image training set.
The authors implemented this through two primary methods:
- CSTM-label: The input image remains unchanged, but the ground-truth depth labels are scaled based on the ratio between the original camera's focal length and the canonical focal length.
- CSTM-image: The input image is physically resized to simulate the field of view of the canonical camera, allowing the model to learn from a consistent geometric perspective.
Random Proposal Normalization Loss (RPNL)
To further improve the precision of the predicted geometry, Metric3D introduces the Random Proposal Normalization Loss (RPNL). Standard depth losses often normalize errors across the entire image, which can lead to the smoothing of fine details. RPNL addresses this by cropping random patches (proposals) from the image and enforcing scale-invariant constraints locally. This forces the model to capture high-frequency geometric structures and sharp boundaries, which are essential for high-quality 3D reconstructions.
Large-Scale Training and Evaluation
The robustness of Metric3D stems from its training on an unprecedented scale. The researchers curated a dataset of over 8 million images by aggregating 11 public datasets, including autonomous driving data (DDAD, Cityscapes, Argoverse2) and general indoor/outdoor scenes (Taskonomy, NYUv2).
The model was evaluated on 8 zero-shot benchmarks—datasets that were entirely excluded from the training process. This rigorous testing ensures that the model is not simply memorizing specific scenes but has learned a generalized understanding of 3D geometry.
Results and Impact
Metric3D demonstrated superior performance across nearly all metrics, winning the 2nd Monocular Depth Estimation Challenge.
- Zero-shot Accuracy: On the NYUv2 benchmark, Metric3D achieved an Absolute Relative Error (AbsRel) of 0.050, significantly outperforming previous state-of-the-art models like MiDaS and LeReS.
- Single-Image Metrology: The model's accuracy is sufficient to perform real-world measurements. For instance, by using the EXIF metadata of a standard photo, the model can accurately estimate the physical dimensions of objects within the scene.
- Enhancing SLAM: When integrated into monocular Simultaneous Localization and Mapping (SLAM) systems, Metric3D effectively mitigates "scale drift." By providing a consistent metric scale, it prevents the virtual map from shrinking or expanding erroneously as the camera moves.
Conclusion
Metric3D represents a significant step toward universal 3D vision. By standardizing the learning process through canonical camera transformation, it proves that metric accuracy and zero-shot generalization are not mutually exclusive. While the current iteration relies on knowing the camera's focal length (often available in EXIF data) to recover absolute scale at inference time, future work aims to integrate focal length estimation directly into the pipeline. This would enable full metric reconstruction for any image, regardless of whether its metadata is preserved.
Summary updated