Camera Radar LiDAR Fusion via Point-Cloud Projection for Multi-Task Learning

Kavin Chandrasekaran, Sorin GrigorescuarXiv 2026 · arXiv

TL;DR

This paper introduces a resource-efficient early fusion architecture that combines camera imagery with projected point clouds from both High-Definition (HD) radar and LiDAR. By performing vehicle detection and free-space segmentation simultaneously in the perspective view, the model achieves state-of-the-art results on the RADIal dataset while maintaining high real-time throughput (over 70 FPS).

What Problem Does This Paper Address?

Reliable perception is the bedrock of autonomous driving, yet every sensor has a "blind spot." Cameras provide rich semantic detail but fail in low light or fog. LiDAR offers high precision but is expensive and can be hampered by heavy precipitation. Conventional radar is robust in all weather but lacks the angular resolution to distinguish fine details.

While many researchers have explored sensor fusion, two major hurdles remain:

  1. Computational Complexity: Processing raw radar data (like ADC signals or Range-Doppler maps) is extremely resource-intensive.
  2. Representation Gaps: Fusing data across different views (e.g., perspective camera view vs. Bird’s-Eye View radar) often requires complex transformations that can introduce errors or lose spatial correlation.

The authors aim to create a unified, multi-task architecture that leverages the strengths of all three sensors without the heavy computational tax of raw data processing.

Methods

The core innovation lies in the Point-Cloud Projection approach. Instead of transforming camera data into a 3D space, the authors project the 3D point clouds from HD radar and LiDAR onto the 2D camera image plane.

  • Early Fusion: The projected point clouds are encoded as binary masks. These masks are concatenated with the 3-channel RGB image, creating a 5-channel input tensor. This "early fusion" ensures the network learns joint representations from the very first layer.
  • Architecture: The model uses an encoder-decoder structure. The encoder consists of a pre-encoder followed by four ResNet-50-style blocks. The decoder progressively recovers spatial resolution using transposed convolutions and skip connections to preserve fine-grained details.
  • Multi-Task Heads: The decoded features feed into two specialized heads:
  • Detection Head: Produces a probability map for vehicle occupancy, optimized using focal loss to handle the extreme imbalance between vehicle and background pixels.
  • Segmentation Head: Estimates drivable vs. non-drivable regions using binary cross-entropy loss.
Fig. 2: Deep Seeing Through Sensors: Early fusion is performed using inputs from (a) the camera, (b) a sparse radar point cloud, and (c) a dense LiDAR point cloud. The encoder comprises a pre-encoder followed by four ResNet-50-style blocks, whose output feature maps are denoted as x0, x1, x2, and x3. A thick purple arrow indicates the connection from the encoder to the decoder, through which the encoded features are upsampled to higher spatial resolutions. The dashed lines indicate skip connections that preserve spatial information. The final outputs are produced by a Detection Head and a Segmentation Head, and the corresponding prediction results are illustrated in Fig. 1 and Fig. 3.

Datasets

The authors primarily utilize the RADIal dataset, which is unique for providing synchronized, high-definition radar, LiDAR, and camera data. They also mention K-Radar and RaDelft as relevant datasets for 4D radar, though RADIal serves as the primary benchmark for this study due to its comprehensive multimodal suite.

Key Insights and Ideas

  • Perspective View is Sufficient: While recent trends favor Bird’s-Eye View (BEV) for fusion, this work demonstrates that staying in the perspective view can be highly effective for front-facing tasks, avoiding the "costly" physical transformations required for BEV.
  • Point Clouds as Efficient Proxies: By using point clouds instead of raw radar tensors, the authors significantly reduce the model's memory footprint and processing time without a major hit to accuracy.
  • Radar as a "Sanity Check": Qualitative results show that radar and LiDAR help the model suppress "false positive" drivable areas (like sidewalks or platforms) that a camera-only model might misclassify due to visual similarity to the road.

Key Results

The proposed fusion model outperformed both standalone modalities and existing fusion frameworks on the RADIal test split:

  • Detection Accuracy: Achieved an Average Precision (AP) of 97.32% and an F1-score of 97.09%, surpassing models like ROFusion and EchoFusion.
  • Segmentation Quality: Reached a mean Intersection-over-Union (mIoU) of 91.77%, a significant jump over the previous state-of-the-art (REFNet++ at 87.58%).
  • Efficiency: The model runs at 70.92 FPS on an NVIDIA RTX A6000, making it one of the fastest fusion models evaluated, with a relatively small GPU memory cost of 1.97 GB.
Fig. 3: Qualitative comparison showing how the fusion model (c) correctly identifies drivable space compared to the camera-only baseline (b).

Limitations and Future Work

The authors acknowledge a few key areas for growth:

  • Binary Focus: The current model only performs binary classification (vehicle vs. background). Future work will aim to extend this to multi-class detection (pedestrians, cyclists, etc.).
  • Weather Analysis: While the sensors used are theoretically robust to weather, this specific study did not explicitly benchmark performance under extreme adverse conditions like heavy snow or torrential rain.
  • Data Scarcity: The field still lacks large-scale, diverse, and precisely annotated multimodal datasets, which limits the ability to train even more robust models.

Summary updated

Abstract

Autonomous vehicles require robust perception systems to operate safely under diverse environmental conditions. While cameras provide high-resolution visual data, their performance significantly degrades in low-light and adverse weather. High Definition (HD) radar offers superior reliability and angular resolution under such conditions but lacks rich visual feature recognition. LiDAR, though expensive, delivers highly precise distance measurements and dense point clouds. Each sensor exhibits unique trade-offs, necessitating the development of advanced sensor fusion algorithms to harness complementary strengths while minimizing computational complexity and maintaining accuracy. In this study, the proposed fusion architecture performs early fusion by combining camera images with perspective-view point cloud data from HD radar and LiDAR sensors. The proposed model is trained to detect vehicles and segment free drivable space simultaneously. Experiments are performed on the RADIal dataset where we demonstrate that fusing point cloud data with camera images in front-view surpasses the performances of the most recent standalone and fusion models.
BibTeX
@misc{chandrasekaran2026camera,
  author = {Kavin Chandrasekaran and Sorin Grigorescu},
  title = {{Camera Radar LiDAR Fusion via Point-Cloud Projection for Multi-Task Learning}},
  year = {2026},
  eprint = {2611.03942},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.03942},
}
Figure