Spatial-Temporal Graph Enhanced DETR Towards Multi-Frame 3D Object Detection

Yifan Zhang, Zhiyu Zhu, Junhui Hou, Dapeng Wu

TL;DR

This paper introduces STEMD, a novel end-to-end framework for multi-frame 3D object detection that enhances the DETR architecture. By incorporating spatial-temporal graph attention and improved query initialization, STEMD achieves state-of-the-art performance, particularly in challenging scenarios like occlusions and fast-moving objects, while maintaining computational efficiency.

What Problem Does This Paper Address?

3D object detection is crucial for applications like autonomous driving. While single-frame 3D detectors have advanced, they struggle with challenges like object occlusion, sparsity, and incompleteness in point cloud data. Multi-frame 3D object detection offers a solution by leveraging temporal information, but existing methods often rely on complex multi-stage pipelines or lack effective modeling of inter-object spatial and temporal relationships. This paper aims to bridge this gap by proposing an end-to-end DETR-based framework that efficiently models these complex relationships for improved multi-frame 3D object detection.

Methods

The proposed STEMD framework builds upon the DETR architecture and introduces three key enhancements:

  • Spatial-Temporal Graph Attention Network (STGA-Net): This novel module models inter-object spatial interactions and temporal dependencies. It represents object queries as nodes in a graph, allowing them to dynamically attend to neighbors. A graph-to-graph attention mechanism further captures temporal dependencies across frames.
  • Temporal Query Recollection (TQR): To mitigate issues where the encoder might miss challenging "hard cases" in the current frame, TQR initializes decoder queries with high-scoring predictions from the previous frame. This provides a stronger starting point for the decoder.
  • IoU Regularization Term: To address the problem of redundant predictions in DETR-like models (which don't use Non-Maximum Suppression), an IoU regularization term is introduced. This term penalizes overlapping bounding boxes, encouraging similar queries to output distinct predictions, especially important in 3D where ground truth boxes in BEV generally don't overlap.

Datasets

The authors evaluate their method on two large-scale datasets:

  • Waymo Open Dataset: A widely used dataset for autonomous driving research, providing LiDAR point clouds, camera images, and object annotations.
  • nuScenes dataset: Another comprehensive dataset for autonomous driving, featuring multi-modal sensor data.

Key Insights and Ideas

  • Graph-based modeling is key for complex interactions: Representing object queries as nodes in a graph and using attention mechanisms allows for a more nuanced understanding of how objects interact spatially and temporally, which is crucial for robust detection.
  • Leveraging past information is vital for hard cases: Initializing current frame detection with successful predictions from the previous frame (TQR) significantly helps in handling occluded or temporarily absent objects.
  • 3D object detection has unique properties: The lack of overlapping ground truth bounding boxes in the Bird's Eye View (BEV) of 3D datasets allows for specific regularization techniques (IoU regularization) that might not be as effective in 2D.
  • End-to-end DETR for multi-frame 3D is promising: The paper demonstrates that a DETR-like architecture can be effectively adapted for multi-frame 3D object detection, offering a more streamlined approach compared to traditional multi-stage methods.

Key Results

STEMD achieves state-of-the-art performance on the Waymo dataset, significantly outperforming previous single-frame and multi-frame detectors. For instance, with an 8-frame input, STEMD achieves a 4.9% improvement in APH on LEVEL_2 for cyclists compared to CenterFormer. Even with a 4-frame input, STEMD surpasses existing methods. The framework also demonstrates strong performance on the nuScenes dataset. Qualitatively, STEMD excels at detecting distant and occluded vehicles and reduces false positives compared to baseline methods. The method is also computationally efficient, with a latency of 115.71 ms for 4 frames on a single GPU, making it suitable for real-time applications.

Limitations and Future Work

The authors acknowledge that STEMD's performance on very fast-moving objects is not as strong as desired. They attribute this to the difficulty of capturing the motion of such objects within the current spatial-temporal modeling. Future work could explore incorporating explicit motion estimation techniques or adaptive frame sampling strategies to better handle fast-moving objects. They also suggest that further advancements in 2D DETR models could be integrated to further improve their 3D detection framework.

Summary updated

Abstract

The Detection Transformer (DETR) has revolutionized the design of CNN-based object detection systems, showcasing impressive performance. However, its potential in the domain of multi-frame 3D object detection remains largely unexplored. In this paper, we present STEMD, a novel end-to-end framework that enhances the DETR-like paradigm for multi-frame 3D object detection by addressing three key aspects specifically tailored for this task. First, to model the inter-object spatial interaction and complex temporal dependencies, we introduce the spatial-temporal graph attention network, which represents queries as nodes in a graph and enables effective modeling of object interactions within a social context. To solve the problem of missing hard cases in the proposed output of the encoder in the current frame, we incorporate the output of the previous frame to initialize the query input of the decoder. Finally, it poses a challenge for the network to distinguish between the positive query and other highly similar queries that are not the best match. And similar queries are insufficiently suppressed and turn into redundant prediction boxes. To address this issue, our proposed IoU regularization term encourages similar queries to be distinct during the refinement. Through extensive experiments, we demonstrate the effectiveness of our approach in handling challenging scenarios, while incurring only a minor additional computational overhead. The code is publicly available at https://github.com/Eaphan/STEMD.
BibTeX
@article{zhang2024spatial,
  author = {Yifan Zhang and Zhiyu Zhu and Junhui Hou and Dapeng Wu},
  title = {{Spatial-Temporal Graph Enhanced DETR Towards Multi-Frame 3D Object Detection}},
  journal = {IEEE Trans. Pattern Anal. Mach. Intell.},
  year = {2024},
  doi = {10.1109/tpami.2024.3443335},
}
Figure