Abstract
Multi-frame aggregation is a cornerstone of modern LiDAR-based 3D object detection, used to densify point clouds and improve detection of distant or sparse objects. However, standard methods apply a fixed aggregation window to the entire scene, leading to a performance trade-off: while stationary objects benefit from more frames, fast-moving objects suffer from motion-induced "smearing" and misalignment. We propose VADet (Variable Aggregation Detection), a data-centric framework that adaptively selects the optimal number of frames for each object based on its estimated speed and point density. Coupled with Random Aggregation Training (RAT), VADet is architecture-agnostic and highly efficient. Our approach achieves state-of-the-art performance on the Waymo Open Dataset, significantly improving detection for fast and sparse objects across multiple baseline architectures.
The Challenge of Fixed Aggregation
In autonomous driving, LiDAR sensors provide sparse 3D snapshots of the environment. Aggregating sequential frames is a common strategy to build a more complete picture of the scene. However, "more data" is not always beneficial. As shown in Figure 1, while stationary objects show a steady increase in Average Precision (AP) as more frames are added, fast-moving objects experience a sharp decline after just a few frames.
This degradation is primarily due to motion blur. When multiple frames are concatenated without perfect motion compensation for every individual dynamic object, the resulting point cloud becomes "smeared" (see Figure 1c), creating distorted shapes that confuse standard 3D detectors.
Variable Aggregation (VADet)
VADet moves away from the "one-size-fits-all" approach by performing aggregation on a per-object basis. The method consists of two main components:
- Adaptive Frame Selection: VADet uses a mapping function, $\eta$, to determine the ideal number of frames for each detected object. This function considers both the object's velocity and its point density. For instance, a distant, sparse, stationary vehicle might be aggregated over 16 frames to maximize shape information, whereas a nearby, dense, fast-moving vehicle might only use 3 frames to maintain geometric clarity.
- Random Aggregation Training (RAT): To avoid the prohibitive cost of training separate models for every possible frame count, the authors introduce RAT. During training, the number of aggregated frames is randomly varied for each sample. This forces the detector to become robust to different levels of point density and temporal context, allowing a single model to handle the variable inputs generated by VADet during inference.
Experimental Results
The researchers evaluated VADet on the Waymo Open Dataset (WOD) using three distinct detector architectures: CenterPoint (voxel-based), VoxelNeXt (fully sparse), and DSVT (transformer-based).
As detailed in Figure 5, VADet consistently outperforms fixed-frame baselines across all categories. Notably, VADet-VoxelNeXt achieved 79.4 L2 AP on the Waymo test set, surpassing significantly more complex state-of-the-art methods like MPPNet. The improvements are most pronounced for "Fast" and "Sparse" objects, where traditional fixed-aggregation methods typically fail to find an optimal balance.
Key Insights
- Density vs. Speed: A critical finding is that speed is not the sole determinant of optimal aggregation. Point density is equally important; sparse objects (often at long ranges) benefit from more frames even at moderate speeds because the added points are necessary to define the object's boundaries.
- Efficiency and Versatility: Unlike multi-stage refinement models that add significant latency (up to 2500ms), VADet operates at the input level, adding only approximately 50ms of overhead. Its architecture-agnostic nature allows it to be integrated into existing detection pipelines with ease.
Limitations and Future Work
While VADet shows impressive gains, it currently relies on an empirical lookup table for the mapping function $\eta$. Future iterations could involve learning this function as a differentiable module. Additionally, while the method is highly effective for vehicles, further tuning may be required for other classes like pedestrians or cyclists, whose motion patterns and point signatures differ significantly. Finally, the system's performance is tied to the accuracy of initial velocity estimates from the first-pass detection.
Summary updated