Multi-modal Streaming 3D Object Detection

Mazen Abdelfattah, Kaiwen Yuan, Z. Jane Wang, Rabab Ward

Abstract

Modern autonomous vehicles rely heavily on mechanical LiDARs for perception. Current perception methods generally require 360° point clouds, collected sequentially as the LiDAR scans the azimuth and acquires consecutive wedge-shaped slices. The acquisition latency of a full scan (~ 100ms) may lead to outdated perception which is detrimental to safe operation. Recent streaming perception works proposed directly processing LiDAR slices and compensating for the narrow field of view (FOV) of a slice by reusing features from preceding slices. These works, however, are all based on a single modality and require past information which may be outdated. Meanwhile, images from high-frequency cameras can support streaming models as they provide a larger FoV compared to a LiDAR slice. However, this difference in FoV complicates sensor fusion. To address this research gap, we propose an innovative camera-LiDAR streaming 3D object detection framework that uses camera images instead of past LiDAR slices to provide an up-to-date, dense, and wide context for streaming perception. The proposed method outperforms prior streaming models on the challenging nuScenes benchmark. It also outperforms powerful full-scan detectors while being much faster. Our method is shown to be robust to missing camera images, narrow LiDAR slices, and small camera-LiDAR miscalibration.

Introduction

Traditional 3D object detection for autonomous vehicles (AVs) typically follows a "wait-and-see" approach. A standard mechanical LiDAR takes approximately 100ms to complete a full 360° rotation. Most detectors wait for this full "sweep" before initiating inference. For a vehicle traveling at 60 km/h, it moves 1.7 meters during a single scan, meaning a 100ms delay can lead to outdated perception—detecting an object where it was, rather than where it currently is.

Streaming detection concept showing the difference between observation and detection time.

Streaming detection addresses this by processing small wedges (slices) of the LiDAR scan as soon as they are acquired. However, a single slice has a very narrow Field of View (FoV), making it difficult to identify large or fragmented objects. While previous methods used temporal memory from past slices, this information is often stale. This paper proposes using real-time camera images to provide the necessary global context.

Methodology

The proposed framework treats camera images as a "global context" provider that is always up-to-date. The system processes LiDAR slices and camera images independently and in parallel to maximize efficiency.

Overview of the multi-modal streaming 3D object detection framework.

The architecture consists of four main components:

  1. Parallel Encoding: The LiDAR branch utilizes a PointPillars encoder, while the camera branch processes images through a dedicated encoder.
  2. Volumetric Image Projection: To bridge the dimensionality gap, 2D image features are projected into a 3D volumetric grid using pinhole camera geometry, which is then collapsed into a Bird’s Eye View (BEV) representation.
  3. Feature Map Cropping: To optimize processing, the model extracts only the relevant wedge of the BEV feature map corresponding to the current LiDAR slice.
  4. Fusion and Detection: BEV features from both modalities are concatenated and processed by a CenterPoint detection head to predict 3D bounding boxes.

Experimental Results

The model was evaluated on the nuScenes dataset. On the benchmark, the 8-slice model achieved a 53.8 mAP (mean Average Precision), outperforming the previous state-of-the-art streaming model, PolarStream (52.9 mAP). Notably, the model demonstrates extreme robustness; even when the LiDAR scan is fragmented into 32 tiny slices, it maintains a 51.1 mAP, whereas LiDAR-only models fail (2.2 mAP). Furthermore, the end-to-end runtime is just 15.8ms, significantly reducing the latency compared to traditional 100ms full-scan methods. The system also shows resilience to sensor miscalibration, maintaining performance despite small rotation or translation errors.

Conclusion

By leveraging camera images as a real-time context provider, this multi-modal streaming framework overcomes the limitations of narrow LiDAR slices and stale temporal memory. The approach provides a significant boost in both accuracy and speed for autonomous vehicle perception. Future research may explore hybrid temporal mechanisms that combine historical LiDAR data with current camera context.

Summary updated

Abstract

Modern autonomous vehicles rely heavily on mechanical LiDARs for perception. Current perception methods generally require 360{\deg} point clouds, collected sequentially as the LiDAR scans the azimuth and acquires consecutive wedge-shaped slices. The acquisition latency of a full scan (~ 100ms) may lead to outdated perception which is detrimental to safe operation. Recent streaming perception works proposed directly processing LiDAR slices and compensating for the narrow field of view (FOV) of a slice by reusing features from preceding slices. These works, however, are all based on a single modality and require past information which may be outdated. Meanwhile, images from high-frequency cameras can support streaming models as they provide a larger FoV compared to a LiDAR slice. However, this difference in FoV complicates sensor fusion. To address this research gap, we propose an innovative camera-LiDAR streaming 3D object detection framework that uses camera images instead of past LiDAR slices to provide an up-to-date, dense, and wide context for streaming perception. The proposed method outperforms prior streaming models on the challenging NuScenes benchmark. It also outperforms powerful full-scan detectors while being much faster. Our method is shown to be robust to missing camera images, narrow LiDAR slices, and small camera-LiDAR miscalibration.
BibTeX
@article{abdelfattah2023multi,
  author = {Mazen Abdelfattah and Kaiwen Yuan and Z. Jane Wang and Rabab Ward},
  title = {{Multi-modal Streaming 3D Object Detection}},
  journal = {IEEE Robotics Autom. Lett.},
  year = {2023},
  doi = {10.1109/lra.2023.3303696},
}
Figure