D-Align: Dual Query Co-attention Network for 3D Object Detection Based on Multi-frame Point Cloud Sequence

Junhyung Lee, Junho Koh, Youngwoo Lee, Jun Won Choi

TL;DR

D-Align is a novel 3D object detector that significantly boosts accuracy by effectively "stitching" together information from a sequence of LiDAR frames. By using a dual-query co-attention network to align and aggregate features across time, it overcomes the "blurring" effect caused by moving objects, outperforming current state-of-the-art methods on the challenging nuScenes benchmark.

What Problem Does This Paper Address?

LiDAR sensors provide a 360-degree 3D view of a robot's surroundings, but a single "sweep" of the laser is often sparse. While a common fix is to merge multiple consecutive sweeps into one large point cloud, this approach has a "diminishing returns" problem.

As you add more frames, moving objects (like cars or cyclists) change position, creating a "ghosting" effect or a smeared distribution of points. This temporal misalignment makes it harder for standard neural networks to pin down an object's exact location and velocity. The authors of D-Align set out to build a model that doesn't just stack frames blindly but intelligently aligns and aggregates them by understanding the motion context between them.

Methods

The core of the paper is the Dual-Query Co-attention Network (DUCANet). Unlike standard detectors that process one feature map, D-Align maintains two sets of "queries":

  1. Target Query Set (T-QS): Represents the features of the current frame we want to detect objects in.
  2. Support Query Set (S-QS): Represents features from previous frames.

The architecture refines these queries through three main components:

  • IDANet (Inter-frame Deformable Alignment Network): This is the "alignment" engine. It uses a Feature Pyramid Motion Extraction (FPMNet) to look at two adjacent frames and estimate how things have moved. It then uses "deformable attention"—a technique that lets the network "reach out" to specific pixels in the previous frame—to warp the support features so they line up perfectly with the target frame.
  • IGANet (Inter-frame Gated Aggregation Network): Once aligned, the support features are merged into the target features. It uses a gated mechanism (essentially a learned filter) to decide which parts of the previous frames are useful and which should be ignored (e.g., due to occlusions).
  • Iterative Refinement: These steps are repeated across multiple layers ($L=3$), progressively sharpening the Bird’s-Eye-View (BEV) features used for the final detection.

Datasets

The researchers evaluated D-Align on the nuScenes dataset, one of the most popular and challenging benchmarks for autonomous driving. It contains 1,000 driving scenes with complex urban traffic and 1.4 million 3D bounding box annotations across 10 object classes.

Key Insights and Ideas

  • Motion Context Matters: Standard deformable attention usually learns offsets from the features themselves. D-Align’s "aha moment" is explicitly feeding a motion-encoding feature (the difference between adjacent frames) into the attention mechanism. This makes the alignment much more robust to fast-moving objects.
  • Dual Queries for Stability: By maintaining separate query sets for target and support frames, the network can iteratively refine the "history" of the scene alongside the "present," leading to more stable spatio-temporal representations.
  • Compensating for Sparsity: The model is particularly effective for classes that usually have very few LiDAR points, such as motorcycles, bicycles, and traffic cones. By "borrowing" points from previous frames and aligning them correctly, the model sees a much denser and clearer object.

Key Results

D-Align was tested using two different "backbone" baselines: PointPillars (D-Align-PP) and CenterPoint (D-Align-CP).

  • State-of-the-Art Performance: D-Align-CP achieved a 70.2 NDS (nuScenes Detection Score) and 64.0 mAP (mean Average Precision) on the test set, outperforming other multi-frame models like STGNN and 3DVID.
  • Significant Gains: Compared to the single-frame PointPillars baseline, D-Align-PP improved the NDS from 45.3% to 63.1%—a massive jump.
  • Ablation Success: Removing the motion-guided alignment (FPMNet) caused a noticeable drop in performance, proving that explicit motion modeling is a key ingredient for multi-frame success.

Limitations and Future Work

  • Temporal Horizon: The authors found that performance peaked at an input length of 3 frames (about 1.5 seconds of data) and actually dipped slightly at 4 frames. This suggests the model currently struggles to maintain coherence over longer durations (2+ seconds) where dynamic changes become too complex.
  • Memory Constraints: Due to the memory-intensive nature of multi-scale attention, the authors were limited to only 3 attention layers.
  • Future Directions: The authors suggest that exploring ways to handle even longer sequences or integrating more efficient attention mechanisms could further push the boundaries of 3D temporal perception.

Summary updated

Abstract

LiDAR sensors are widely used for 3D object detection in various mobile robotics applications. LiDAR sensors continuously generate point cloud data in real-time. Conventional 3D object detectors detect objects using a set of points acquired over a fixed duration. However, recent studies have shown that the performance of object detection can be further enhanced by utilizing spatio-temporal information obtained from point cloud sequences. In this paper, we propose a new 3D object detector, named D-Align, which can effectively produce strong bird's-eye-view (BEV) features by aligning and aggregating the features obtained from a sequence of point sets. The proposed method includes a novel dual-query co-attention network that uses two types of queries, including target query set (T-QS) and support query set (S-QS), to update the features of target and support frames, respectively. D-Align aligns S-QS to T-QS based on the temporal context features extracted from the adjacent feature maps and then aggregates S-QS with T-QS using a gated fusion mechanism. The dual queries are updated through multiple attention layers to progressively enhance the target frame features used to produce the detection results. Our experiments on the nuScenes dataset show that the proposed D-Align method greatly improved the performance of a single frame-based baseline method and significantly outperformed the latest 3D object detectors. Code is available at https://github.com/junhyung-SPALab/D-Align.
BibTeX
@inproceedings{lee2023d,
  author = {Junhyung Lee and Junho Koh and Youngwoo Lee and Jun Won Choi},
  title = {{D-Align: Dual Query Co-attention Network for 3D Object Detection Based on Multi-frame Point Cloud Sequence}},
  booktitle = {IEEE International Conference on Robotics and Automation},
  year = {2023},
  url = {https://openreview.net/forum?id=vn3Mq8DVaK},
}

Loading related content...

Figure