Cost-Aware Evaluation and Model Scaling for LiDAR-Based 3D Object Detection

Xiaofang Wang, Kris M. Kitani

TL;DR

This paper highlights a critical flaw in 3D object detection research: new architectures are often compared to older baselines without controlling for computational cost (latency). By simply scaling up SECOND, a classic and "simple" one-stage detector, the authors show it can match or outperform modern, highly complex state-of-the-art models like PV-RCNN++ when given the same "latency budget."

What Problem Does This Paper Address?

In the world of 2D image recognition (like ImageNet), it is standard practice to compare models within the same "compute regime" (e.g., comparing models with similar FLOPs or parameters). However, the authors observe that in LiDAR-based 3D object detection, this discipline is largely missing.

Recent papers often claim superior performance based on novel, complex architectural components (like sophisticated point-voxel fusion or multi-stage refinement). But because these new models are often much slower than the baselines they compare against, it’s unclear if their gains come from "smarter" design or simply from using more "horsepower." This paper aims to level the playing field through cost-aware evaluation.

Methods

The authors take SECOND, a well-known, grid-based one-stage detector, and systematically scale it up to see how much performance is left on the table. They explore three primary scaling dimensions:

  1. Network Depth: Increasing the number of layers.
  2. Network Width: Increasing the number of channels.
  3. Pre-head Resolution: Increasing the spatial dimension of the feature map right before it enters the detection head.

Key Technical Innovations: * Pre-head Resolution Scaling: They find that reducing downsampling in the 3D backbone (keeping a higher resolution feature map) is more effective than just adding depth or width. This allows for a denser sampling of object anchors. * Residual Connections: They note that simply deepening 3D sparse convolutional networks leads to optimization issues. Adding residual connections (res versions) is essential to unlock the potential of deeper backbones. * Head Comparison: They evaluate both traditional Anchor-based heads and more recent Center-based heads (from CenterPoint) across different scales.

Datasets

The study exclusively uses the Waymo Open Dataset (WOD), currently the largest and most rigorous public benchmark for LiDAR-based 3D object detection. They utilize both the full training set and a "20% mini-set" for efficient ablation studies.

Key Insights and Ideas

  • The "Simple" Baseline is a Giant: A properly scaled SECOND model (dubbed A1res or A2res) easily outperforms many complex methods published in the last year. It can even match the performance of the current state-of-the-art, PV-RCNN++, if both are allowed the same inference latency.
  • Resolution is King: In 3D detection, the size of an object doesn't change with distance (unlike 2D images). Therefore, the primary benefit of higher resolution isn't "seeing small things" better, but rather providing a denser grid for anchors and keypoints to land on, which significantly boosts localization accuracy.
  • Diminishing Returns of Complexity: The performance gap between simple one-stage detectors and complex two-stage detectors shrinks significantly as the backbone size increases.
  • The "Center Head" Advantage Flips: While Center-based heads are much better than Anchor-based heads on small backbones, the paper finds that on very large backbones, the Anchor-based head can actually catch up or even slightly overtake the Center head in certain metrics.

Key Results

  • Efficiency: SECOND-Anchor-A1res achieves a similar Mean Average Precision weighted by Heading (mAPH) to PV-RCNN++ (Anchor) but is 2x faster (67ms vs. ~130ms+).
  • Performance: Scaled SECOND models outperformed recent specialized methods like VoTr-TSD and Pyramid-PV in both speed and accuracy.
  • Scaling Strategy: The authors demonstrate that "compound scaling" (increasing depth, width, and resolution simultaneously) is generally superior to scaling a single dimension, echoing findings from 2D vision (like EfficientNet).

Limitations and Future Work

  • Scope of Baselines: The paper focuses almost entirely on scaling the SECOND architecture. While SECOND is a foundational component of many other models, it remains to be seen if these scaling laws apply identically to "range-view" or purely "point-based" detectors.
  • Hardware Specificity: Latency was measured on a specific setup (RTX 3090 GPU). Since sparse convolutions perform differently across hardware generations, the "cost" might shift on edge devices or different GPU architectures.
  • Call to Action: The authors strongly recommend that the 3D detection community adopt cost-aware benchmarks and include "Scaled SECOND" as a standard baseline to ensure that future "architectural innovations" are truly providing value beyond just increased computation.

Summary updated

Abstract

Considerable research effort has been devoted to LiDAR-based 3D object detection and empirical performance has been significantly improved. While progress has been en-couraging, we observe an overlooked issue: it is not yet common practice to compare different 3D detectors under the same cost, e.g., inference latency. This makes it difficult to quantify the true performance gain brought by recently proposed architecture designs. The goal of this work is to conduct a cost-aware evaluation of LiDAR-based 3D object detectors. Specifically, we focus on SECOND, a simple grid-based one-stage detector, and analyze its performance under different costs by scaling its original architecture. Then we compare the family of scaled SECOND with recent 3D detection methods, such as Voxel R-CNN and PV-RCNN++. The results are surprising. We find that, if allowed to use the same latency, SECOND can match the performance of PV-RCNN++, the current state-of-the-art method on the Waymo Open Dataset. Scaled SECOND also easily outperforms many recent 3D detection methods published during the past year. We recommend future research control the inference cost in their empirical comparison and include the family of scaled SECOND as a strong baseline when presenting novel 3D detection methods.
BibTeX
@inproceedings{wang2023cost,
  author = {Xiaofang Wang and Kris M. Kitani},
  title = {{Cost-Aware Evaluation and Model Scaling for LiDAR-Based 3D Object Detection}},
  booktitle = {IEEE International Conference on Robotics and Automation},
  year = {2023},
  doi = {10.1109/icra48891.2023.10161165},
}
Figure