DynStatF: An Efficient Feature Fusion Strategy for LiDAR 3D Object Detection

Yao Rong, Xiangyu Wei, Tianwei Lin, Yueyu Wang, Enkelejda Kasneci

Abstract

LiDAR-based 3D object detection is a cornerstone of autonomous driving, yet it faces a fundamental trade-off between point cloud density and spatial precision. While aggregating multiple LiDAR sweeps increases point density and provides temporal context, it introduces "motion blur" that degrades localization accuracy. This paper presents DynStatF (Dynamic-Static Fusion), a dual-pathway architecture designed to resolve this conflict. By fusing a multi-frame "dynamic" branch for rich semantics with a single-frame "static" branch for precise spatial anchoring, DynStatF achieves state-of-the-art results on the nuScenes benchmark. Our approach utilizes Neighborhood Cross Attention (NCA) and Dynamic-Static Interaction (DSI) to efficiently align and fuse these features, providing significant performance gains over baseline models with minimal computational overhead.

Introduction

In the pursuit of reliable autonomous navigation, LiDAR sensors are indispensable for their high-precision depth perception. However, a single LiDAR scan (or "sweep") is often too sparse to accurately classify distant or small objects. To mitigate this, standard practice involves temporal aggregation—stacking multiple consecutive sweeps into a single input. While this process enriches the semantic context and reveals motion patterns, it introduces the "motion blur" problem.

Motion blur occurs when the movement of objects or slight inaccuracies in the ego-vehicle's pose estimation cause aggregated points to smear across space. This blurring makes it difficult for detection heads to identify the exact centers and boundaries of objects. Conversely, relying solely on a single frame preserves sharp spatial boundaries but lacks the density required for robust recognition. DynStatF addresses this by treating the single-frame data as a spatial anchor, using it to "pull" and align the rich information from multi-frame sequences into a precise coordinate space.

Methodology

Dual-Pathway Architecture

DynStatF employs a dual-pathway design that processes multi-frame (dynamic) and single-frame (static) inputs concurrently. The dynamic branch typically processes 10 LiDAR sweeps (approximately 0.5s of data) to capture motion and shape, while the static branch processes only the current sweep. To maintain efficiency, the static branch is designed to be lightweight, utilizing a simple Voxel Feature Encoder (VFE) rather than a computationally expensive 3D backbone.

Neighborhood Cross Attention (NCA)

Standard Transformer-style global attention is often prohibitively expensive for Bird’s Eye View (BEV) feature maps due to their high dimensionality. Furthermore, global attention is largely redundant in this context; an object's position in the current frame is highly correlated with its position in the previous frames.

To exploit this spatial locality, the authors introduce Neighborhood Cross Attention (NCA). NCA restricts the attention mechanism to a local $k \times k$ neighborhood. In this module, the static features serve as the queries, while the dynamic features serve as keys and values. This configuration ensures that the multi-frame semantic information is filtered and aligned based on the precise spatial coordinates of the current frame, effectively "de-blurring" the temporal data.

Dynamic-Static Interaction (DSI)

As features progress through the 2D backbone and become denser, the model employs the Dynamic-Static Interaction (DSI) module. Unlike the attention-based NCA, DSI is a CNN-based module that performs comprehensive pixel-wise interaction between the two branches. This late-stage fusion ensures that the final feature map passed to the detection head contains a harmonious blend of high-resolution spatial details and deep temporal context.

Experimental Results

Dataset and Setup

The framework was evaluated on the nuScenes dataset, which includes 1,000 video sequences of complex urban driving. The model was tested against 10 object classes using the NuScenes Detection Score (NDS) and mean Average Precision (mAP) as primary metrics.

Performance Gains

DynStatF demonstrates significant "plug-and-play" improvements when integrated into existing detection architectures:

  • PointPillars: Integration of DynStatF improved the PointPillars baseline from 57.7% to 61.6% NDS, and from 44.6% to 50.5% mAP.
  • CenterPoint: When applied to the more robust CenterPoint backbone, the framework achieved 67.7% NDS and 61.0% mAP, establishing a new state-of-the-art for non-ensemble methods at the time of publication.

Qualitative analysis confirms that DynStatF is particularly effective in crowded environments. It excels at distinguishing individual pedestrians in close proximity and accurately defining the bounding boxes of fast-moving vehicles, which are typically the most affected by temporal blurring.

Discussion and Insights

A key finding of the ablation studies was the necessity of the dual-pathway approach. Models that attempted to share weights between the static and dynamic branches (a "Single" pathway) performed significantly worse. This indicates that multi-frame and single-frame data represent fundamentally different feature spaces—one optimized for "what" an object is, and the other for "where" it is—requiring specialized extraction logic for each.

Conclusion and Future Work

DynStatF provides an elegant solution to the motion blur problem in LiDAR detection by intelligently fusing temporal richness with spatial precision. While the model significantly improves detection in dynamic scenes, it still faces challenges with extreme occlusions where objects are almost entirely hidden. Future research directions include the integration of camera data into the dual-pathway architecture to leverage visual semantic cues and exploring further pruning techniques to reduce the computational footprint of the static branch even further.

Summary updated

Abstract

Augmenting LiDAR input with multiple previous frames provides richer semantic information and thus boosts performance in 3D object detection, However, crowded point clouds in multi-frames can hurt the precise position information due to the motion blur and inaccurate point projection. In this work, we propose a novel feature fusion strategy, DynStaF (Dynamic-Static Fusion), which enhances the rich semantic information provided by the multi-frame (dynamic branch) with the accurate location information from the current single-frame (static branch). To effectively extract and aggregate complimentary features, DynStaF contains two modules, Neighborhood Cross Attention (NCA) and Dynamic-Static Interaction (DSI), operating through a dual pathway architecture. NCA takes the features in the static branch as queries and the features in the dynamic branch as keys (values). When computing the attention, we address the sparsity of point clouds and take only neighborhood positions into consideration. NCA fuses two features at different feature map scales, followed by DSI providing the comprehensive interaction. To analyze our proposed strategy DynStaF, we conduct extensive experiments on the nuScenes dataset. On the test set, DynStaF increases the performance of PointPillars in NDS by a large margin from 57.7% to 61.6%. When combined with CenterPoint, our framework achieves 61.0% mAP and 67.7% NDS, leading to state-of-the-art performance without bells and whistles.
BibTeX
@inproceedings{rong2023dynstatf,
  author = {Yao Rong and Xiangyu Wei and Tianwei Lin and Yueyu Wang and Enkelejda Kasneci},
  title = {{DynStatF: An Efficient Feature Fusion Strategy for LiDAR 3D Object Detection}},
  booktitle = {Workshops @ CVPR},
  year = {2023},
}
Figure