Multi-View Fusion of Sensor Data for Improved Perception and Prediction in Autonomous Driving

Sudeep Fadadu, Shreyash Pandey, Darshan Hegde, Yi Shi, Fang-Chieh Chou, Nemanja Djuric, Carlos Vallespi-GonzalezWACV 2022

TL;DR

This paper introduces a novel multi-view fusion architecture that combines Bird's-Eye View (BEV) and Range-View (RV) representations of LiDAR data, along with camera and HD map information, to improve object detection and trajectory prediction for autonomous driving. By leveraging the strengths of both BEV (spatial constancy) and RV (fine-grained detail), their method achieves state-of-the-art performance on challenging datasets with minimal computational overhead.

What Problem Does This Paper Address?

Autonomous driving systems rely heavily on accurately detecting objects in their environment and predicting their future movements. Traditional approaches often process sensor data (like LiDAR, cameras, and HD maps) using separate methods or by projecting them into a single representation. Bird's-Eye View (BEV) is popular for its ability to maintain object size consistency and simplify fusion with HD maps, but it can lose fine-grained details from LiDAR. Range-View (RV), on the other hand, preserves this detail and is efficient for fusing camera data, but can make object size vary with range and requires learning complex transformations to BEV. This paper addresses the challenge of effectively combining the complementary benefits of these different sensor representations to achieve more robust and accurate perception and prediction.

Methods

The core of this work is a multi-view fusion architecture that processes LiDAR data in both BEV and RV formats.

  • Multi-View LiDAR Encoding: LiDAR point clouds are encoded into two distinct representations:
  • BEV: Historical LiDAR sweeps are voxelized into a top-down grid, capturing occupancy. This aligns with existing state-of-the-art methods like MultiXNet.
  • RV: The current LiDAR sweep is processed into a dense 2D image, preserving fine-grained range, intensity, and positional information.
  • Sensor Fusion:
    • RV Fusion: Camera RGB images are fused with the current LiDAR RV features using a lightweight network. This fusion is performed in the RV space, which is less sensitive to projection distortions than BEV.
  • BEV Fusion: The processed BEV LiDAR features and HD map features are combined. The fused RV features (from LiDAR and camera) are then projected into the BEV space and fused with the BEV LiDAR and map features.
  • Unified Network: The fused multi-view features are then fed into a backbone network (based on MultiXNet) for joint object detection and trajectory prediction. The authors propose a generic point-based feature projection method to transfer features between views.
  • Loss Function: The model uses a combination of focal loss for object detection and smooth L1 regression loss for trajectory prediction, with a decay factor for future waypoints.

Datasets

The proposed method is evaluated on two datasets:

  • ATG4D: A proprietary dataset collected by Aurora Innovation, featuring data from a 64-beam LiDAR and a front camera. It includes over 1 million frames from 5,500 scenes with 3D bounding box labels up to 100m range.
  • nuScenes: A public dataset with a 32-beam LiDAR and front camera, containing 1,000 scenes and 390,000 LiDAR sweeps.

The authors note that ATG4D is larger and richer, providing more statistically significant results, especially for rare classes like pedestrians and bicyclists.

Key Insights and Ideas

  • Complementary Strengths: The paper strongly advocates for combining BEV and RV representations, highlighting that BEV excels at spatial consistency and HD map fusion, while RV captures crucial fine-grained details, especially for smaller objects.
  • Efficient RV-Camera Fusion: Performing camera-LiDAR fusion in the RV space is shown to be more robust to calibration errors and computationally efficient compared to direct BEV projection.
  • Generalizability: The proposed multi-view fusion framework is designed to be general and can be applied to improve other existing BEV or RV-based algorithms, not just MultiXNet.
  • Impact on Small Objects: The improvements are most pronounced for smaller objects like pedestrians and bicyclists, underscoring the value of RV's detailed information.

Key Results

  • Improved Detection and Prediction: The proposed LC-MV-Lite method (LiDAR + Camera Multi-View) significantly outperforms the MultiXNet baseline on the ATG4D dataset across all actor classes for both detection (AP) and prediction (DE). For instance, it shows a 5.5% improvement in pedestrian AP and a 6.5% improvement in bicyclist AP on nuScenes compared to MultiXNet.
  • Competitive Latency: While there is a slight increase in latency (e.g., from 35ms to 43ms on ATG4D for LC-MV-Lite), it remains within real-time requirements for typical SDV applications. The proposed fusion method adds significantly less latency compared to other fusion techniques like ContFuse.
  • Qualitative Improvements: Figure 2 demonstrates qualitative examples where the proposed method successfully detects objects (occluded vehicles, pedestrians, bicyclists) that the baseline misses, with accurate trajectory predictions for these newly detected actors.
  • Camera Fusion Benefits: Metrics within the camera's Field of View (FOV) show even larger gains, indicating the camera's contribution is most impactful in areas where LiDAR might be sparse.

Limitations and Future Work

  • Latency Trade-off: While efficient, the multi-view approach does introduce some computational overhead compared to single-view methods. The paper notes that using a heavier backbone like ResNet-34 for camera features (LC-MV-ResNet) leads to higher latency.
  • Rare Classes: The paper acknowledges that performance on rare classes like bicyclists can be more unstable, potentially due to limited training data for these classes.
  • Future Work: The authors suggest that the framework is generalizable and can be extended to use multiple cameras (e.g., side and rear cameras) to further enhance perception capabilities while maintaining real-time performance. They also imply that further exploration of RV fusion for historical sweeps could be beneficial, though they opted against it in this work due to complexity concerns.

Summary updated

BibTeX
@inproceedings{fadadu2022multi,
  author = {Sudeep Fadadu and Shreyash Pandey and Darshan Hegde and Yi Shi and Fang-Chieh Chou and Nemanja Djuric and Carlos Vallespi-Gonzalez},
  title = {{Multi-View Fusion of Sensor Data for Improved Perception and Prediction in Autonomous Driving}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2022},
}
Figure