MonoJSG: Joint Semantic and Geometric Cost Volume for Monocular 3D Object Detection

Qing Lian, Peiliang Li, Xiaozhi Chen

Abstract

Due to the inherent ill-posed nature of 2D-3D projection, monocular 3D object detection lacks accurate depth recovery ability. Although the deep neural network (DNN) enables monocular depth-sensing from high-level learned features, the pixel-level cues are usually omitted due to the deep convolution mechanism. To benefit from both the powerful feature representation in DNN and pixel-level geometric constraints, we reformulate the monocular object depth estimation as a progressive refinement problem and propose a joint semantic and geometric cost volume to model the depth error. Specifically, we first leverage neural networks to learn the object position, dimension, and dense normalized 3D object coordinates. Based on the object depth, the dense coordinates patch together with the corresponding object features is reprojected to the image space to build a cost volume in a joint semantic and geometric error manner. The final depth is obtained by feeding the cost volume to a refinement network, where the distribution of semantic and geometric error is regularized by direct depth supervision. Through effectively mitigating depth error by the refinement framework, we achieve state-of-the-art results on both the KITTI and Waymo datasets.

Introduction

Estimating the 3D position of an object from a single 2D image is a fundamentally "ill-posed" problem. Because a single image lacks parity or depth data, a car that appears small in the image could either be a small car nearby or a large car far away. Current monocular detection approaches generally fall into two categories: feature-based methods that use Deep Neural Networks (DNNs) to regress depth directly, and geometry-based methods that use 2D-3D constraints like sparse keypoints. While feature-based methods are powerful, they often ignore pixel-level geometric consistency. Conversely, geometry-based methods are mathematically sound but sensitive to noise and occlusion due to their reliance on sparse data. MonoJSG addresses these limitations by utilizing dense, pixel-level cues—combining semantic features with geometric constraints—to refine depth estimates.

Methodology

The MonoJSG framework operates in two primary stages: a proposal stage and a refinement stage.

Proposal Stage

The model utilizes a modified CenterNet architecture to predict standard 2D and 3D attributes, including object category, 2D bounding box, dimensions, orientation, and an initial depth estimate. A critical component of this stage is the estimation of Normalized Object Coordinates (NOC), which provide a dense 3D map of the object's surface relative to its center.

Refinement Stage

The core innovation is the Joint Semantic and Geometric Cost Volume. Using the initial depth estimate, the model reprojects the dense 3D coordinates back onto the 2D image plane. It then constructs a cost volume by comparing original image pixels with reprojected pixels. Rather than relying on raw photometric data, which can be unreliable in textureless regions like windshields, MonoJSG compares learned semantic features. To optimize computation, the cost volume uses adaptive sampling based on the network's uncertainty. A 3D CNN then processes this 4D cost volume to identify the minimum error point, yielding a refined depth value.

The MonoJSG framework consists of a proposal stage for initial estimates and a refinement stage using an adaptive cost volume.

Analysis of the Error Landscape

A key insight of MonoJSG is the superiority of semantic features over photometric ones for geometric refinement. Learned features are more robust against distractions like reflections. Furthermore, by using dense coordinates rather than sparse keypoints, the system becomes more resilient to partial occlusions.

Comparison of depth error landscapes: MonoJSG provides a smoother, more convex loss surface compared to sparse keypoints or photometric methods.

As shown in the error distribution analysis, the joint cost volume creates a smoother and more convex landscape compared to sparse keypoint methods. This convexity significantly aids the optimizer in converging to the true depth, effectively mitigating localization errors that are often visible in Bird's Eye View (BEV) projections.

Experimental Results

MonoJSG was evaluated on the KITTI and Waymo Open Datasets, achieving state-of-the-art performance at the time of publication:

  • KITTI: On the "Car" category (Moderate difficulty), it achieved 16.14% AP in 3D detection, outperforming previous geometry-based methods by nearly 2%.
  • Waymo: The model showed a 26.42% improvement in 3D mAP (at IoU=0.5) compared to the second-best approach.
  • Efficiency: Despite the complexity of a 4D cost volume, the model maintains real-time efficiency with an inference time of 42ms.

Conclusion and Future Work

While MonoJSG significantly improves monocular 3D detection, it faces challenges in extreme occlusion or low-visibility scenarios where initial NOC estimation may fail. Additionally, errors in initial dimension or orientation estimates can propagate through the refinement stage. Future research may explore incorporating temporal information from video sequences to further stabilize coordinate estimation and depth refinement.

Summary updated

Abstract

Due to the inherent ill-posed nature of 2D-3D projection, monocular 3D object detection lacks accurate depth recovery ability. Although the deep neural network (DNN) enables monocular depth-sensing from high-level learned features, the pixel-level cues are usually omitted due to the deep convolution mechanism. To benefit from both the pow-erful feature representation in DNN and pixel-level geomet-ric constraints, we reformulate the monocular object depth estimation as a progressive refinement problem and propose a joint semantic and geometric cost volume to model the depth error. Specifically, we first leverage neural networks to learn the object position, dimension, and dense normal-ized 3D object coordinates. Based on the object depth, the dense coordinates patch together with the corresponding object features is reprojected to the image space to build a cost volume in a joint semantic and geometric error man-ner. The final depth is obtained by feeding the cost volume to a refinement network, where the distribution of semantic and geometric error is regularized by direct depth supervision. Through effectively mitigating depth error by the re-finement framework, we achieve state-of-the-art results on both the KITTI and Waymo datasets. <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">1</sup> <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">1</sup> Code available at https://github.com/lianqingll/MonoJSG
BibTeX
@misc{lian2022monojsg,
  author = {Qing Lian and Peiliang Li and Xiaozhi Chen},
  title = {{MonoJSG: Joint Semantic and Geometric Cost Volume for Monocular 3D Object Detection}},
  year = {2022},
  eprint = {2203.08563},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2203.08563},
}
Figure