Abstract
Due to the inherent ill-posed nature of 2D-3D projection, monocular 3D object detection lacks accurate depth recovery ability. Although the deep neural network (DNN) enables monocular depth-sensing from high-level learned features, the pixel-level cues are usually omitted due to the deep convolution mechanism. To benefit from both the powerful feature representation in DNN and pixel-level geometric constraints, we reformulate the monocular object depth estimation as a progressive refinement problem and propose a joint semantic and geometric cost volume to model the depth error. Specifically, we first leverage neural networks to learn the object position, dimension, and dense normalized 3D object coordinates. Based on the object depth, the dense coordinates patch together with the corresponding object features is reprojected to the image space to build a cost volume in a joint semantic and geometric error manner. The final depth is obtained by feeding the cost volume to a refinement network, where the distribution of semantic and geometric error is regularized by direct depth supervision. Through effectively mitigating depth error by the refinement framework, we achieve state-of-the-art results on both the KITTI and Waymo datasets.
Introduction
Estimating the 3D position of an object from a single 2D image is a fundamentally "ill-posed" problem. Because a single image lacks parity or depth data, a car that appears small in the image could either be a small car nearby or a large car far away. Current monocular detection approaches generally fall into two categories: feature-based methods that use Deep Neural Networks (DNNs) to regress depth directly, and geometry-based methods that use 2D-3D constraints like sparse keypoints. While feature-based methods are powerful, they often ignore pixel-level geometric consistency. Conversely, geometry-based methods are mathematically sound but sensitive to noise and occlusion due to their reliance on sparse data. MonoJSG addresses these limitations by utilizing dense, pixel-level cues—combining semantic features with geometric constraints—to refine depth estimates.
Methodology
The MonoJSG framework operates in two primary stages: a proposal stage and a refinement stage.
Proposal Stage
The model utilizes a modified CenterNet architecture to predict standard 2D and 3D attributes, including object category, 2D bounding box, dimensions, orientation, and an initial depth estimate. A critical component of this stage is the estimation of Normalized Object Coordinates (NOC), which provide a dense 3D map of the object's surface relative to its center.
Refinement Stage
The core innovation is the Joint Semantic and Geometric Cost Volume. Using the initial depth estimate, the model reprojects the dense 3D coordinates back onto the 2D image plane. It then constructs a cost volume by comparing original image pixels with reprojected pixels. Rather than relying on raw photometric data, which can be unreliable in textureless regions like windshields, MonoJSG compares learned semantic features. To optimize computation, the cost volume uses adaptive sampling based on the network's uncertainty. A 3D CNN then processes this 4D cost volume to identify the minimum error point, yielding a refined depth value.
Analysis of the Error Landscape
A key insight of MonoJSG is the superiority of semantic features over photometric ones for geometric refinement. Learned features are more robust against distractions like reflections. Furthermore, by using dense coordinates rather than sparse keypoints, the system becomes more resilient to partial occlusions.
As shown in the error distribution analysis, the joint cost volume creates a smoother and more convex landscape compared to sparse keypoint methods. This convexity significantly aids the optimizer in converging to the true depth, effectively mitigating localization errors that are often visible in Bird's Eye View (BEV) projections.
Experimental Results
MonoJSG was evaluated on the KITTI and Waymo Open Datasets, achieving state-of-the-art performance at the time of publication:
- KITTI: On the "Car" category (Moderate difficulty), it achieved 16.14% AP in 3D detection, outperforming previous geometry-based methods by nearly 2%.
- Waymo: The model showed a 26.42% improvement in 3D mAP (at IoU=0.5) compared to the second-best approach.
- Efficiency: Despite the complexity of a 4D cost volume, the model maintains real-time efficiency with an inference time of 42ms.
Conclusion and Future Work
While MonoJSG significantly improves monocular 3D detection, it faces challenges in extreme occlusion or low-visibility scenarios where initial NOC estimation may fail. Additionally, errors in initial dimension or orientation estimates can propagate through the refinement stage. Future research may explore incorporating temporal information from video sequences to further stabilize coordinate estimation and depth refinement.
Summary updated