Survey and Systematization of 3D Object Detection Models and Methods

Moritz Drobnitzky, Jonas Friederich, Bernhard Egger, Patrick Zschech

Introduction

While 2D object detection has reached a high level of maturity, often surpassing human performance, it remains insufficient for real-world spatial applications such as autonomous driving and robotics. These systems require a comprehensive three-dimensional understanding of their environment, specifically the "6 Degrees of Freedom"—the precise location, size, and orientation of objects in 3D space.

The field of 3D object detection (3DOD) has seen a surge in research, resulting in a fragmented landscape of specialized methods. Many existing surveys focus on specific niches, such as LiDAR-only data or autonomous driving contexts. This paper addresses the need for a holistic systematization, covering the entire 3DOD pipeline to help researchers navigate design choices and understand their impact on performance.

The 3DOD Pipeline: A Systematic Framework

The core contribution of this work is a structured framework that decomposes the 3DOD process into four major stages. This taxonomy allows any model to be classified based on its specific implementation at each step.

The four-stage 3D object detection pipeline systematized in the survey.
  1. Input Data: Raw sensory inputs are categorized into monocular RGB images, 2.5D RGB-D images, and true 3D Point Clouds.
  2. Data Representation: This stage defines how raw data is structured for processing. Common methods include converting point clouds into Volumetric Grids (voxels), Projections (such as Bird’s Eye View or Range View), or maintaining Point-wise structures.
  3. Feature Extraction: The transition from raw data to meaningful descriptors. The survey distinguishes between traditional Handcrafted features and modern Deep Learning approaches, including CNNs, PointNets, and Sparse Convolutions.
  4. Detection Module: The final architecture that outputs 3D bounding boxes. This includes the choice between Single-Stage and Two-Stage detectors, as well as Anchor-based versus Anchorless techniques.

Benchmarking 3DOD: Datasets

The evolution of 3DOD is closely tied to the development of robust benchmarks. The paper reviews several key datasets across different domains:

Comparison of major 3D object detection datasets across autonomous driving and indoor domains.
  • Autonomous Driving: KITTI remains a classic pioneer, while Waymo Open offers massive scale and diversity. nuScenes provides multimodal data with full 360° coverage.
  • Indoor Scenes: Datasets like NYUv2 and SUN RGB-D focus on cluttered environments where object occlusion and stacking are common challenges.
  • Object-Centric: Google’s Objectron dataset features video clips with sparse point cloud representations, focusing on individual object geometry.

Technical Insights and Trade-offs

The survey identifies several fundamental tensions in 3DOD design:

  • Representation vs. Computation: Voxelization provides the structure needed for standard CNNs but often leads to information loss and high memory consumption. Conversely, Bird’s Eye View (BEV) projections are efficient but struggle with vertical overlaps and small objects.
  • The Power of PointNet: A major milestone was the introduction of PointNet, which demonstrated that deep learning could process raw, unordered point clouds directly using symmetric functions to achieve permutation invariance.
  • Handling Sparsity: Unlike dense images, 3D point clouds are mostly empty space. Sparse Convolutions have emerged as a critical solution, skipping empty voxels to drastically reduce computational load.
  • Multi-modal Fusion: There is a clear trend toward Feature Fusion, combining the depth accuracy of LiDAR with the rich texture of cameras to overcome the limitations of single-modality systems.

The analysis of over 100 papers reveals a dominant shift toward end-to-end deep learning, replacing handcrafted features. Efficiency has also improved significantly; modern architectures like PointPillars now achieve the frame rates necessary for real-time autonomous vehicle operation. Furthermore, there is a growing preference for Anchorless methods (e.g., VoteNet) which are naturally better suited to the sparse nature of 3D data than traditional 2D-inspired anchor boxes.

Despite these advances, challenges remain. Discretization still leads to spatial information loss, and real-time processing on edge devices is difficult due to high data dimensionality. Future research is expected to focus on Transformer-based architectures for point clouds and more sophisticated Non-linear Multi-modal Fusion techniques to better handle sensor discrepancies and noise.

Summary updated

BibTeX
@article{drobnitzky2024survey,
  author = {Moritz Drobnitzky and Jonas Friederich and Bernhard Egger and Patrick Zschech},
  title = {{Survey and Systematization of 3D Object Detection Models and Methods}},
  journal = {The Visual Computer},
  year = {2024},
  doi = {10.1007/s00371-023-02891-1},
}
Figure