Multi-Sensor 3D Object Box Refinement for Autonomous Driving

Peiliang Li, Siqi Liu, Shaojie ShenarXiv 2019 · arXiv

Introduction

Accurate 3D object detection is a cornerstone of autonomous driving, yet different sensors present conflicting strengths and data formats. Monocular cameras are cost-effective and semantically rich but poor at depth estimation. Stereo cameras and LiDAR provide superior depth but represent data as dense pixel grids and sparse point clouds, respectively.

This paper introduces a flexible 3D object detection framework that addresses these challenges by using a monocular camera for initial predictions and "plugging in" stereo or LiDAR sensors for high-precision refinement. The core innovation is a unified "instance vector" representation that maps raw sensor data to an object's local 3D geometry, allowing for a shared mathematical approach to refining object locations across different sensor types.

Methodology

The authors propose a two-stage system designed for modularity and geometric consistency:

  1. Monocular Proposal: A lightweight detector (based on Faster R-CNN) takes a single image and predicts 2D boxes, object classes, and initial 3D properties (dimensions, orientation, and a coarse depth).
  2. Unified Refinement: If stereo or LiDAR data is available, the system refines the initial 3D box using a novel Instance Vector Representation.
The proposed two-stage framework: monocular proposals followed by modular refinement using stereo or LiDAR data.

The Instance Vector

The instance vector acts as a "Rosetta Stone" for sensors. For every observed element (a pixel in an image or a point in a cloud), the network predicts its 3D coordinate relative to the object's local coordinate frame, normalized to a [0, 1] range. This allows the system to translate disparate inputs into the same local geometric space.

Visualization of the Instance Vector mapping observed elements to a normalized local 3D coordinate frame.
  • Stereo Refinement: The system predicts instance vectors for pixels in the left image. It then performs "Photometric Alignment," warping the object's shape from the left image to the right image and finding the 3D position that minimizes the visual difference.
  • LiDAR Refinement: The system predicts instance vectors for raw LiDAR points. It then performs "Point Cloud Alignment," minimizing the distance between the predicted local shape and the actual observed points.

Key Results and Insights

The framework was evaluated on the KITTI Object Benchmark, the industry standard for autonomous driving perception.

  • Monocular SOTA: The base monocular detector achieved 16.02% AP (Average Precision) for cars in the "Easy" category, outperforming previous state-of-the-art methods.
  • Massive Stereo Gains: Adding stereo refinement boosted performance dramatically to 61.96% AP, surpassing other stereo-based methods like Stereo R-CNN.
  • Shape-Awareness: Unlike methods that treat objects as simple boxes, the instance vector allows the model to learn actual surface shapes (e.g., wheels, tail-lights). This makes alignment more precise because the model "knows" which part of the object it is observing.
  • Modular Flexibility: The system is truly "plug-and-play." It functions as a pure monocular detector but becomes significantly more accurate as additional sensors are integrated, without requiring a full architectural retrain.

Limitations and Future Work

While robust, the refinement stage is currently limited by the quality of the initial monocular 2D proposal; if the monocular detector misses an object, the refinement modules never activate. Additionally, predicting instance vectors and running optimization loops adds computational overhead. The authors suggest that incorporating depth information earlier in the pipeline, such as in the Region Proposal Network (RPN) stage, could improve system recall and overall accuracy.

Summary updated

Abstract

We propose a 3D object detection system with multi-sensor refinement in the context of autonomous driving. In our framework, the monocular camera serves as the fundamental sensor for 2D object proposal and initial 3D bounding box prediction. While the stereo cameras and LiDAR are treated as adaptive plug-in sensors to refine the 3D box localization performance. For each observed element in the raw measurement domain (e.g., pixels for stereo, 3D points for LiDAR), we model the local geometry as an instance vector representation, which indicates the 3D coordinate of each element respecting to the object frame. Using this unified geometric representation, the 3D object location can be unified refined by the stereo photometric alignment or point cloud alignment. We demonstrate superior 3D detection and localization performance compared to state-of-the-art monocular, stereo methods and competitive performance compared with the baseline LiDAR method on the KITTI object benchmark.
BibTeX
@misc{li2019multi,
  author = {Peiliang Li and Siqi Liu and Shaojie Shen},
  title = {{Multi-Sensor 3D Object Box Refinement for Autonomous Driving}},
  year = {2019},
  eprint = {1909.04942},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1909.04942},
}
Figure