Multi-Modal Streaming 3D Object Detection

Mazen Abdelfattah, Kaiwen Yuan, Z. Jane Wang, Rabab Ward

TL;DR

This paper introduces the first multi-modal framework for streaming 3D object detection, which processes LiDAR data in small "slices" as they arrive rather than waiting for a full 360° scan. By replacing outdated "memory" from past LiDAR slices with real-time, wide-context camera images, the model achieves state-of-the-art accuracy with significantly lower latency, making it ideal for high-speed autonomous driving.

What Problem Does This Paper Address?

Traditional 3D object detection for autonomous vehicles (AVs) is a "wait-and-see" game. A standard mechanical LiDAR takes about 100ms to complete a full 360° rotation. Most detectors wait for this full "sweep" before starting inference. In a world where a car traveling at 60 km/h moves 1.7 meters during that single scan, a 100ms delay can lead to "outdated perception"—detecting an object where it was, not where it is.

"Streaming" detection aims to solve this by processing small wedges (slices) of the LiDAR scan as soon as they are acquired. However, a single slice has a very narrow Field of View (FoV), making it hard to identify large or fragmented objects. Previous streaming methods tried to fix this by using "memory" (features from past slices), but this information is often stale and inaccurate in dynamic environments. Furthermore, existing sensor fusion methods (combining LiDAR and cameras) usually require the two sensors to have perfectly overlapping FoVs, which isn't the case when you're looking at a tiny 1/32nd slice of a LiDAR scan.

Methods

The authors propose a framework that treats camera images as a "global context" provider that is always up-to-date.

  1. Parallel Encoding: The system processes LiDAR slices and camera images independently and in parallel. The LiDAR branch uses a standard PointPillars encoder.
  2. Volumetric Image Projection: To bridge the gap between 2D images and 3D space, the model projects 2D image features into a 3D volumetric grid using pinhole camera geometry. This grid is then collapsed into a Bird’s Eye View (BEV) representation.
  3. Feature Map Cropping: Since a LiDAR slice only covers a small portion of the environment, processing a full-sized BEV map is wasteful. The authors introduce a "crop and stack" method: they extract only the relevant wedge of the feature map, stack multiple wedges into a mini-batch for efficient GPU processing, and then "uncrop" them before the final detection.
  4. Fusion and Detection: The BEV features from both the camera and LiDAR are concatenated and fed into a CenterPoint detection head, which predicts 3D bounding boxes without needing predefined "anchor boxes."

Datasets

The model was evaluated on the nuScenes dataset, a large-scale public benchmark for autonomous driving that provides synchronized data from a 32-beam LiDAR and six cameras covering a full 360° view.

Key Insights and Ideas

  • Images as Context, Not Just Detail: Instead of using images just to "paint" points with color, this model uses them to provide the spatial context that a narrow LiDAR slice lacks. This allows the model to detect an object even if the current LiDAR slice only hits a tiny corner of it.
  • Decoupling for Robustness: By encoding modalities separately before fusing them in BEV space, the model is much more robust to miscalibration. Traditional "point-wise" fusion (where every LiDAR point is mapped to a specific pixel) fails if the sensors are even slightly out of alignment. This model’s volumetric approach is far more forgiving.
  • Efficiency via Parallelism: Because the camera and LiDAR branches don't depend on each other during encoding, they can run simultaneously on hardware (GPUs/FPGAs), drastically reducing the "bottleneck" effect of sequential processing.

Key Results

  • Superior Accuracy: On the nuScenes benchmark, their 8-slice model achieved a 53.8 mAP (mean Average Precision), outperforming the previous best streaming model, PolarStream (52.9 mAP), and even some full-scan detectors.
  • Extreme Streaming: Even when the LiDAR scan was broken into 32 tiny slices (extreme fragmentation), the model maintained a high mAP of 51.1, whereas LiDAR-only models saw their performance collapse to near zero (2.2 mAP).
  • Speed: The end-to-end runtime for an 8-slice model is just 15.8ms. This is significantly faster than the 100ms "wait time" required for traditional full-scan detectors.
  • Calibration Resilience: The model showed only a minor drop in performance even with a 1° rotation or 10cm translation error in sensor calibration, a scenario that typically breaks other fusion-based detectors.

Limitations and Future Work

While the model is robust, it still relies on the availability of camera data. The authors noted a performance dip when the back camera was "missing," though it still outperformed LiDAR-only baselines.

The paper focuses on replacing "past LiDAR memory" with "current camera context," but future work could explore if a more sophisticated temporal mechanism (perhaps a transformer-based memory) could combine the best of both worlds—using both up-to-date images and historical LiDAR data—without the "staleness" issues identified here. Additionally, testing on even more diverse weather conditions (where cameras might struggle) would be a valuable next step.

Summary updated

Abstract

Modern autonomous vehicles rely heavily on mechanical LiDARs for perception. Current perception methods generally require <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$360^\circ$</tex-math></inline-formula> point clouds, collected sequentially as the LiDAR scans the azimuth and acquires consecutive wedge-shaped slices. The acquisition latency of a full scan ( <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$\sim\!\text{100}\;\text{ms}$</tex-math></inline-formula> ) may lead to outdated perception which is detrimental to safe operation. Recent streaming perception works proposed directly processing LiDAR slices and compensating for the narrow field of view (FOV) of a slice by reusing features from preceding slices. These works, however, are all based on a single modality and require past information which may be outdated. Meanwhile, images from high-frequency cameras can support streaming models as they provide a larger FoV compared to a LiDAR slice. However, this difference in FoV complicates sensor fusion. We propose an innovative camera-LiDAR streaming 3D object detection framework that uses camera images instead of past LiDAR slices to provide an up-to-date, dense, and wide context for streaming perception. The proposed method outperforms prior streaming models and powerful full-scan baselines on the challenging NuScenes benchmark in detection accuracy and end-to-end runtime. Our method is shown to be robust to missing camera images, narrow LiDAR slices, and small camera-LiDAR miscalibration.
BibTeX
@article{abdelfattah2023multi,
  author = {Mazen Abdelfattah and Kaiwen Yuan and Z. Jane Wang and Rabab Ward},
  title = {{Multi-Modal Streaming 3D Object Detection}},
  journal = {IEEE Robotics Autom. Lett.},
  year = {2023},
  doi = {10.1109/lra.2023.3303696},
}

Loading related content...

Figure