Abstract
Most 3D reconstruction methods may only recover scene properties up to a global scale ambiguity. We present a novel approach to single view metrology that can recover the absolute scale of a scene represented by 3D heights of objects or camera height above the ground as well as camera parameters of orientation and field of view, using just a monocular image acquired in unconstrained condition. Our method relies on data-driven priors learned by a deep network specifically designed to imbibe weakly supervised constraints from the interplay of the unknown camera with 3D entities such as object heights, through estimation of bounding box projections. We leverage categorical priors for objects such as humans or cars that commonly occur in natural images, as references for scale estimation. We demonstrate state-of-the-art qualitative and quantitative results on several datasets as well as applications including virtual object insertion. Furthermore, the perceptual quality of our outputs is validated by a user study.
Introduction
In computer vision, reconstructing a 3D scene from a single 2D image is an inherently ill-posed problem due to scale ambiguity: a small object close to the camera can appear identical to a large object far away. While deep learning has improved relative depth estimation, recovering absolute measurements—such as the exact height of a person or the camera's elevation—typically requires specialized hardware like LiDAR or known reference objects.
This paper introduces ScaleNet, a framework designed for "Single View Metrology in the Wild." ScaleNet recovers the absolute 3D scale of a scene from a single unconstrained monocular image by leveraging categorical priors, such as the known average heights of humans and cars, to act as "statistical rulers."
Methodology
ScaleNet combines geometric camera calibration with scale estimation through an end-to-end architecture. The system is built upon several key components:
- Camera Calibration: A module estimates the camera's pitch and vertical field-of-view (FOV). Predicting FOV rather than focal length directly provides greater stability for unconstrained images.
- Object Estimation: The network detects common objects (humans and cars), predicting their 2D bounding boxes and keypoints.
- Geometry-Based Scale Estimation: Instead of estimating height directly from pixels, a PointNet-inspired module uses 2D geometric features—specifically bounding box coordinates and their vertical offset from the horizon—to estimate camera height ($h_{cam}$).
- Weakly Supervised Reprojection Loss: The model is trained to minimize the difference between detected 2D boxes and the 2D projections of estimated 3D objects. This ensures that the recovered 3D geometry is consistent with the observed image.
A critical insight is the use of the "Upright Ratio." By predicting how much a person is crouching or sitting using keypoints, the model can adjust its height expectations, making the system robust to non-standing poses.
Experiments and Results
The authors evaluated ScaleNet across several diverse datasets, including KITTI for autonomous driving metrics, SUN360 for calibration, and IMDB-23K for human height validation.
- Accuracy: ScaleNet achieved a mean camera height error of just 0.12m on the KITTI dataset, significantly outperforming previous probabilistic graphical models.
- Refinement: A cascade refinement process iteratively reduces reprojection errors, demonstrating that the geometric constraints successfully guide the model toward a consistent solution.
Virtual Object Insertion
One of the most compelling applications of this technology is realistic virtual object insertion. By accurately recovering the scene's scale and camera parameters, virtual objects can be placed with correct perspective and relative sizing.
In user studies, participants found ScaleNet’s insertions more realistic than baseline methods nearly 60% of the time, validating the perceptual quality of the recovered scale.
Conclusion and Future Work
ScaleNet demonstrates that absolute scale can be recovered from single images by combining deep learning with classical geometric constraints. However, the current model assumes a flat ground plane, which limits its application in environments with varying elevations like stairs or hills. Future research may focus on "amodal" reasoning to estimate the full height of partially occluded objects and extending the framework to handle non-planar ground surfaces.
Summary updated