LiDAR-Based 3D Temporal Object Detection via Motion-Aware LiDAR Feature Fusion

Gyuhee Park, Junho Koh, Jisong Kim, Jun Moon, Jun Won ChoiSensors 2024 · OpenReview

TL;DR

This paper introduces TM3DOD, a novel LiDAR-based 3D object detection method that leverages temporal information from consecutive LiDAR scans. By incorporating a Temporal Voxel Encoder (TVE) and a Motion-Aware Feature Aggregation Network (MFANet), TM3DOD significantly improves detection accuracy and robustness, especially for dynamic objects, outperforming single-frame baselines and achieving state-of-the-art performance on the nuScenes benchmark.

What Problem Does This Paper Address?

Autonomous driving systems rely heavily on accurate 3D object detection to perceive their surroundings. While many existing LiDAR-based 3D object detectors perform well on individual point cloud frames, they often overlook the rich temporal information present in sequential LiDAR data. This temporal data, which captures motion and changes over time, holds significant potential to enhance detection capabilities, particularly in challenging scenarios involving occlusions, missing points, and irregular sampling. The paper aims to bridge this gap by developing a method that effectively utilizes this temporal information to improve the accuracy and robustness of 3D object detection.

Methods

The proposed TM3DOD method consists of two main novel components:

  • Temporal Voxel Encoder (TVE): This module addresses the temporal relationships within LiDAR data at the voxel level. It processes points from consecutive temporal laser sweeps within a 3D voxel. The TVE employs a Temporal Channel Attention Network (TCANet), which assigns varying attention weights to points based on their temporal relationships. This allows the encoder to focus on crucial temporal information and important channels within the point features, making it more robust to noisy or irrelevant points. The TCANet uses temporal-wise attention (TWA) and channel-wise attention (CWA) to extract relevant features.
  • Motion-Aware Feature Aggregation Network (MFANet): This module enhances the Bird's-Eye View (BEV) feature representation by incorporating temporal information. It quantifies the temporal variation between consecutive BEV feature maps to generate motion features. These motion features are then integrated with the current BEV feature map. The MFANet uses a two-step process: first, it calculates the difference between adjacent BEV feature maps and passes them through a convolutional network to extract motion features. Then, it applies spatial-wise attention (SWA) and channel-wise attention (CWA) to these motion features to highlight important spatial regions and channels. Finally, these enhanced motion features are added to the corresponding BEV feature map to create a more discriminative representation.

These modules are integrated with a standard LiDAR backbone network for feature extraction and a 3D detection head for final object prediction.

Datasets

The primary dataset used for training and evaluation in this paper is nuScenes. This dataset is a popular benchmark for 3D object detection in autonomous driving, featuring a large collection of annotated driving scenes with LiDAR point clouds. The authors used the nuScenes dataset for both their main experiments and ablation studies.

Key Insights and Ideas

  • Temporal Information is Crucial: The paper strongly emphasizes that neglecting temporal information in LiDAR data is a significant missed opportunity for improving 3D object detection. By explicitly modeling temporal dependencies, substantial performance gains can be achieved.
  • Motion Matters for Detection: Quantifying and incorporating motion cues (temporal variations) into feature representations is key to enhancing the detection of dynamic objects, which are often challenging for single-frame methods.
  • Attention Mechanisms Enhance Temporal Modeling: The use of temporal and channel-wise attention in both the voxel and BEV domains effectively filters and prioritizes relevant temporal and spatial information, leading to more robust feature representations.
  • Voxel-Level Temporal Encoding is Beneficial: The TVE module demonstrates that capturing temporal relationships at the voxel level, before BEV projection, can provide a richer representation that benefits downstream tasks.

Key Results

The proposed TM3DOD method demonstrated significant improvements:

  • On nuScenes Benchmark: When integrated with the CenterPoint baseline, TM3DOD achieved a 3.61% increase in mean Average Precision (mAP) and a 2.64% increase in nuScenes Detection Score (NDS) compared to the single-frame CenterPoint.
  • Ablation Studies:
    • Adding the MFANet module to a single-frame baseline (PointPillar) resulted in a 2.19% gain in mAP and a 1.32% gain in NDS.
  • Adding both TVE and MFANet modules resulted in a 4.18% gain in mAP and a 2.40% gain in NDS over the single-frame baseline.
  • The runtime increased by a modest 13 ms and 8 ms for MFANet and TVE respectively, indicating efficient temporal integration.
  • Qualitative Results: Figure 4 shows that TM3DOD successfully detects both static and moving objects, whereas the single-frame baseline struggled with moving objects.

Limitations and Future Work

  • Performance on Small/Stationary Objects: The authors acknowledge that TM3DOD, like many multi-frame methods, still faces challenges in improving performance for small or stationary objects such as pedestrians, traffic cones, and barriers. These categories exhibit similar LiDAR data characteristics, making them difficult to distinguish even with temporal information.
  • Future Work: The authors suggest sensor fusion as a promising direction to overcome the limitations in detecting these challenging object categories. Combining LiDAR data with other sensor modalities (e.g., cameras) could provide complementary information to improve detection accuracy for these specific cases.

Summary updated

Abstract

Recently, the growing demand for autonomous driving in the industry has led to a lot of interest in 3D object detection, resulting in many excellent 3D object detection algorithms. However, most 3D object detectors focus only on a single set of LiDAR points, ignoring their potential ability to improve performance by leveraging the information provided by the consecutive set of LIDAR points. In this paper, we propose a novel 3D object detection method called temporal motion-aware 3D object detection (TM3DOD), which utilizes temporal LiDAR data. In the proposed TM3DOD method, we aggregate LiDAR voxels over time and the current BEV features by generating motion features using consecutive BEV feature maps. First, we present the temporal voxel encoder (TVE), which generates voxel representations by capturing the temporal relationships among the point sets within a voxel. Next, we design a motion-aware feature aggregation network (MFANet), which aims to enhance the current BEV feature representation by quantifying the temporal variation between two consecutive BEV feature maps. By analyzing the differences and changes in the BEV feature maps over time, MFANet captures motion information and integrates it into the current feature representation, enabling more robust and accurate detection of 3D objects. Experimental evaluations on the nuScenes benchmark dataset demonstrate that the proposed TM3DOD method achieved significant improvements in 3D detection performance compared with the baseline methods. Additionally, our method achieved comparable performance to state-of-the-art approaches.
BibTeX
@inproceedings{park2024lidar,
  author = {Gyuhee Park and Junho Koh and Jisong Kim and Jun Moon and Jun Won Choi},
  title = {{LiDAR-Based 3D Temporal Object Detection via Motion-Aware LiDAR Feature Fusion}},
  booktitle = {Sensors},
  year = {2024},
  url = {https://openreview.net/forum?id=WPOanwGCGf},
}

Loading related content...

Figure