Abstract
Most 3D reconstruction methods may only recover scene properties up to a global scale ambiguity. We present a novel approach to single view metrology that can recover the absolute scale of a scene represented by 3D heights of objects or camera height above the ground as well as camera parameters of orientation and field of view, using just a monocular image acquired in unconstrained conditions. Our method relies on data-driven priors learned by a deep network specifically designed to imbibe weakly supervised constraints from the interplay of the unknown camera with 3D entities such as object heights, through estimation of bounding box projections. We leverage categorical priors for objects such as humans or cars that commonly occur in natural images, as references for scale estimation.
Introduction: The Challenge of Scale Ambiguity
In computer vision, reconstructing a 3D scene from a single 2D image is an ill-posed problem due to scale ambiguity. Without a reference, it is impossible to distinguish between a small object close to the camera and a large object far away. Traditional single-view metrology requires manual input of a reference object of known size to resolve this ambiguity.
This paper introduces ScaleNet, a framework designed to automate this process by using ubiquitous objects—specifically humans and cars—as implicit "metric rulers." By leveraging the statistical distribution of heights for these categories, the system can calibrate the camera and measure the scene without human intervention.
Methods: ScaleNet Architecture
ScaleNet operates in two primary stages:
- Initial Calibration and Detection: A Mask R-CNN backbone estimates camera intrinsics (focal length and pitch) and detects 2D bounding boxes for reference objects. It also predicts an initial 3D height for these objects.
- Cascade Refinement: A series of refinement layers take 2D bounding box coordinates and the estimated horizon line to iteratively update camera height ($h_{cam}$) and object heights ($h_{obj}$).
The model is trained using a reprojection loss, ensuring that estimated 3D heights and camera parameters project accurately back into the observed 2D bounding boxes. To handle varied poses, the network predicts an "upright ratio" based on keypoints, allowing it to apply standing height priors even to sitting or crouching individuals.
Experimental Evaluation and Results
The authors evaluated ScaleNet on several datasets, including KITTI for absolute camera height and IMDB-23K for person height prediction. ScaleNet significantly outperformed traditional Probabilistic Graphical Model (PGM) baselines. On the KITTI dataset, the multi-category version achieved a mean camera height error of just 0.12m, compared to 0.14m for the baseline.
Beyond raw metrics, the system enables practical applications like virtual object insertion. In user studies, participants found ScaleNet’s insertions significantly more realistic in terms of scale and perspective compared to baseline methods, validating the perceptual quality of the absolute scale recovery.
Conclusion and Future Work
ScaleNet demonstrates that categorical priors can effectively resolve scale ambiguity in "in the wild" monocular images. While the current model assumes a flat ground plane and is optimized for urban environments, it represents a significant step toward automated, metric-accurate 3D scene understanding. Future work includes addressing non-flat terrain and incorporating amodal reasoning to handle occlusions.
Summary updated