Leveraging Temporal Information for 3D Detection and Domain Adaptation

Cunjun Yu, Zhongang Cai, Daxuan Ren, Haiyu ZhaoarXiv 2020 · arXiv

Introduction

LiDAR sensors serve as the primary perception modality for autonomous vehicles, yet they are fundamentally constrained by point cloud sparsity. As objects recede from the sensor, the density of laser returns diminishes rapidly, complicating the task of shape recognition and orientation estimation. Historically, many 3D detection frameworks—influenced by the single-sweep nature of the KITTI dataset—have treated the world as a sequence of independent snapshots. This paper challenges that paradigm by demonstrating how simple temporal concatenation can significantly enhance detection performance and provide vital motion cues.

Methodology

The researchers propose a streamlined pipeline to integrate temporal data into existing 3D detection architectures. The approach consists of four key components:

  1. Multi-Frame Concatenation: Instead of processing a single frame, the model aggregates the current sweep with the three preceding frames. By transforming these points into a unified ego-vehicle coordinate system, the resulting point cloud is substantially densified.
  2. Temporal Encoding: To prevent the model from perceiving moving objects as elongated, blurred masses, a fifth attribute—a relative timestamp—is appended to the standard $(x, y, z, \text{intensity})$ point features. This allows the neural network to differentiate between the current position of an object and its historical locations.
  3. Base Architecture: The study utilizes the Part $A^2$ detector from the OpenPCDet framework. Hyperparameters, including voxel size and point cloud range, were optimized to accommodate the larger scale and density of modern datasets like the Waymo Open Dataset (WOD).
  4. Data Interoperability: To leverage existing KITTI-based tools, the authors developed a conversion utility to map WOD sequences into the KITTI format, facilitating standardized training and evaluation.

The "Trail" Effect and Temporal Insights

The primary advantage of temporal integration is the emergence of "trails" left by moving objects. For pedestrians, who often appear as sparse vertical clusters of points in a single frame, these trails act as a definitive indicator of heading.

Visualizing the 'trail' effect in multi-frame point clouds, which provides critical heading cues for moving objects.

Beyond motion cues, temporal information serves as a form of "super-resolution" for stationary or slow-moving objects. By accumulating points from different perspectives over several frames, the model constructs a more complete 3D geometric representation. However, the authors emphasize that timestamps are mandatory; without them, the network struggles to interpret the elongated point clusters of fast-moving vehicles, often resulting in inaccurate bounding box dimensions.

Experimental Results

Evaluation on the Waymo Open Dataset demonstrates that the inclusion of temporal information yields consistent improvements across all object categories.

Performance comparison showing significant mAP gains across all classes when temporal information is included.

Key performance metrics include:

  • Vehicle Detection: mAP increased from 0.6465 to 0.6640.
  • Pedestrian Detection: mAP saw a substantial jump from 0.6371 to 0.6600.
  • Heading Accuracy: The gap between standard mAP and mAPH (which incorporates heading error) narrowed significantly, particularly for pedestrians, validating the utility of temporal trails for orientation prediction.

Additionally, the team found that a simple post-processing step—removing bounding boxes that contain no points from the current frame—provided a marginal but consistent boost to final accuracy.

Domain Adaptation and Future Directions

While temporal information provides a clear performance ceiling lift, the authors note that Domain Adaptation (DA) remains a significant challenge. Experiments involving "naive fine-tuning" on target domains revealed that a model trained in one environment does not easily generalize to another, even with temporal cues. The performance gap between source and target domains remains wide, suggesting that more sophisticated, unsupervised, or semi-supervised adaptation techniques are required. Future work may also explore extending the temporal window beyond four frames to capture longer-term dependencies in complex urban environments.

Summary updated

Abstract

Ever since the prevalent use of the LiDARs in autonomous driving, tremendous improvements have been made to the learning on the point clouds. However, recent progress largely focuses on detecting objects in a single 360-degree sweep, without extensively exploring the temporal information. In this report, we describe a simple way to pass such information in the learning pipeline by adding timestamps to the point clouds, which shows consistent improvements across all three classes.
BibTeX
@misc{yu2020leveraging,
  author = {Cunjun Yu and Zhongang Cai and Daxuan Ren and Haiyu Zhao},
  title = {{Leveraging Temporal Information for 3D Detection and Domain Adaptation}},
  year = {2020},
  eprint = {2006.16796},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2006.16796},
}
Figure