A Multimodal Hybrid Late-Cascade Fusion Network for Enhanced 3D Object Detection

Carlo Sgaravatti, Roberto Basla, Riccardo Pieroni, Matteo Corno, Sergio M. Savaresi, Luca Magri, Giacomo Boracchi

TL;DR

This paper introduces a hybrid late-cascade fusion framework that significantly improves 3D object detection by combining independent LiDAR and stereo RGB camera networks. By leveraging geometric and epipolar constraints, the system filters out LiDAR false positives and recovers missed detections (false negatives) for challenging, sparse objects like pedestrians and cyclists—all without requiring complex end-to-end retraining.

What Problem Does This Paper Address?

In autonomous driving, safely navigating roads requires accurately detecting objects in 3D space. LiDAR sensors provide excellent geometric precision but struggle to detect small, distant, or occluded objects (like pedestrians and cyclists) because the resulting point clouds are highly sparse. Conversely, RGB cameras capture rich semantic details but lack direct depth information.

Existing multimodal fusion methods attempt to bridge this gap but come with distinct trade-offs:

  • Early fusion combines raw features early on, which demands perfectly aligned data, limits data augmentation, and introduces heavy computational overhead.
  • Cascade fusion uses 2D images to generate 3D proposals, making the system entirely dependent on the 2D detector's accuracy.
  • Late fusion runs parallel networks and combines their outputs, but traditionally only focuses on filtering out false positives, failing to recover objects completely missed by the LiDAR sensor.

The authors tackle this dilemma by proposing a hybrid approach that combines the best of late and cascade fusion to maximize object recall while keeping the system flexible and computationally efficient.

Fig. 1: Left: LiDAR branch struggles in detecting cyclists and pedestrians. Center: RGB branch correctly detects all the objects but lacks 3D information. Right: Our method recovers all the detections missed by the LiDAR and provides 3D information.

Methods

The proposed architecture consists of five distinct modules that operate on top of pre-trained, single-modal detectors:

  1. RGB and LiDAR Branches: The system runs a 2D RGB detector (Faster R-CNN) on stereo images and a 3D LiDAR detector on the point cloud. Crucially, the authors lower the confidence threshold and relax the Non-Maxima Suppression (NMS) of the LiDAR branch to intentionally boost initial recall, accepting that it will produce more false positives.
  2. Bounding Box Matching (Late Fusion): 3D bounding boxes from the LiDAR branch are projected onto the 2D image planes. An optimization problem (solved via the Jonker-Volgenant algorithm) matches these projections with 2D RGB detections based on Intersection over Union (IoU). Unmatched LiDAR boxes are discarded as false positives.
  3. Detection Recovery (Cascade Fusion): For 2D RGB detections that find no matching LiDAR box, the system initiates a recovery process. It uses epipolar geometry to pair unmatched 2D boxes across the left and right stereo views, intersects their 3D viewing frustums, and crops the corresponding sparse point cloud. A lightweight Frustum Localizer (FrustumPointNet) then estimates the 3D bounding box from this tightly constrained region.
  4. Semantic Fusion: Finally, a probabilistic ensemble framework resolves any class mismatches between the modalities, favoring the superior semantic classification of the RGB images to update labels and confidence scores.
The 5-module hybrid late-cascade fusion architecture.

Datasets

The approach was evaluated on the popular KITTI object detection benchmark, which consists of 7,481 training samples and 7,518 testing samples. The authors utilized the standard training/validation split (3,712 training, 3,769 validation) to benchmark performance across three difficulty levels (Easy, Moderate, Hard) for Cars, Pedestrians, and Cyclists.

Key Insights and Ideas

  • Strategic Over-Detection: Intentionally lowering the detection bar for the LiDAR branch allows the network to capture weak geometric signals from difficult objects. The system relies safely on the 2D image matching phase to prune away the resulting noise.
  • Stereo Frustum Intersection: While traditional frustum methods project a single 2D box into a massive 3D cone, intersecting frustums from stereo views drastically narrows down the 3D search space. This makes it significantly easier to isolate and localize highly sparse point clusters.
  • True Plug-and-Play Modularization: Because the fusion occurs at the decision/output level, the framework can be plugged on top of virtually any state-of-the-art 2D or 3D single-modal detector without needing to retrain the underlying models from scratch.

Key Results

The hybrid framework was tested using several prominent LiDAR baselines (SECOND, PointPillars, PartA2, and PV-RCNN).

  • Massive Recall Gains for Small Objects: When applied to PointPillars, the method boosted Pedestrian 3D Average Precision (AP) by +13.11% (Easy), +12.98% (Moderate), and +11.69% (Hard). Cyclist 3D AP improved by +11.11% in moderate scenarios.
  • Maintained Car Accuracy: Performance on cars remained stable with marginal improvements, which is ideal since LiDAR detectors are already highly optimized for large, distinct geometric shapes like vehicles.
  • Real-Time Efficiency: When paired with a fast encoder like PointPillars, the entire pipeline achieves an inference speed of 29.7 FPS on a single NVIDIA A100 GPU, comfortably exceeding the standard 10–20 FPS frame rate of modern autonomous vehicle sensors.

Limitations and Future Work

  • Stereo Dependency: The Detection Recovery module relies inherently on a stereo camera setup to perform epipolar matching and frustum intersection. It cannot be directly deployed on vehicles equipped only with monocular cameras.
  • 2D Bottleneck: The recovery mechanism is strictly bounded by the performance of the 2D detector. If the RGB network fails to detect a pedestrian in both images, the system has no way to recover the missing 3D object.
  • Future Directions: The authors suggest extending the framework to multi-camera cross-view configurations and incorporating temporal tracking to smooth out object trajectories over consecutive frames.

Summary updated

Abstract

We present a new way to detect 3D objects from multimodal inputs, leveraging both LiDAR and RGB cameras in a hybrid late-cascade scheme, that combines an RGB detection network and a 3D LiDAR detector. We exploit late fusion principles to reduce LiDAR False Positives, matching LiDAR detections with RGB ones by projecting the LiDAR bounding boxes on the image. We rely on cascade fusion principles to recover LiDAR False Negatives leveraging epipolar constraints and frustums generated by RGB detections of separate views. Our solution can be plugged on top of any underlying single-modal detectors, enabling a flexible training process that can take advantage of pre-trained LiDAR and RGB detectors, or train the two branches separately. We evaluate our results on the KITTI object detection benchmark, showing significant performance improvements, especially for the detection of Pedestrians and Cyclists.
BibTeX
@inproceedings{sgaravatti2024multimodal,
  author = {Carlo Sgaravatti and Roberto Basla and Riccardo Pieroni and Matteo Corno and Sergio M. Savaresi and Luca Magri and Giacomo Boracchi},
  title = {{A Multimodal Hybrid Late-Cascade Fusion Network for Enhanced 3D Object Detection}},
  booktitle = {W-CODA @ ECCV},
  year = {2024},
  url = {https://openreview.net/forum?id=ad6Q54fHMr},
}
Figure