Abstract
Radar-camera 3D object detection aims at interacting radar signals with camera images for identifying objects of interest and localizing their corresponding 3D bounding boxes. To overcome the severe sparsity and ambiguity of radar signals, we propose a robust framework based on probabilistic denoising diffusion modeling. We design our framework to be easily implementable on different multi-view 3D detectors without the requirement of using LiDAR point clouds during either the training or inference. In specific, we first design our framework with a denoised radar-camera encoder via developing a lightweight denoising diffusion model with semantic embedding. Secondly, we develop the query denoising training into 3D space via introducing the reconstruction training at depth measurement for the transformer detection decoder. Our framework achieves new state-of-the-art performance on the nuScenes 3D detection benchmark but with few computational cost increases compared to the baseline detectors.
Introduction
In autonomous driving, 3D object detection is a critical task that typically relies on a multi-sensor suite. While LiDAR provides precise geometry, its high cost limits widespread adoption. Cameras offer rich semantic detail but struggle with depth and velocity estimation. Radar presents a cost-effective alternative, excelling in velocity and distance measurement even in adverse weather. However, radar data is inherently noisy, sparse, and lacks height information, often leading to "ghost" reflections.
Existing fusion methods frequently use LiDAR as a supervisory signal during training to interpret radar data. Others face "semantic loss" when projecting 2D image features into a 3D Bird’s-Eye-View (BEV). This research introduces a "LiDAR-free" framework that leverages Probabilistic Denoising Diffusion Models (DDMs) to robustly fuse radar and camera data, effectively handling radar's inherent noise.
Methodology
The proposed framework, referred to as DR (Denoising Radar-camera), consists of two primary components designed to clean and integrate multi-modal sensor data.
Denoised Radar-Camera Encoder
Instead of simple feature concatenation, the framework employs a Denoising Diffusion Model. Unlike standard diffusion models that start from pure noise, this sampling process begins with the original noisy radar features. Visual context from camera images is injected as semantic embeddings into the diffusion process. This guidance allows the model to distinguish between legitimate objects and noise based on visual cues (e.g., identifying a car-shaped region in the image to validate a radar return).
3D Query Denoising
The framework utilizes a Transformer-based decoder, common in DETR-like detectors. An auxiliary "query denoising" task is introduced at the 3D level. By adding noise to ground-truth 3D boxes and training the model to reconstruct the original depth and coordinates, the training process is stabilized, and spatial associations between sensors are strengthened.
The framework is designed as a "plug-and-play" module and has been successfully integrated with major baseline detectors such as BEVDet, PETR, and BEVFormer.
Results and Discussion
The model was evaluated on the nuScenes benchmark, the industry standard for multi-sensor fusion. The results demonstrate that semantic guidance is crucial; applying a standard diffusion model without cross-modal context yielded minimal gains.
Performance Gains
The DR framework achieved significant improvements across all baseline models. For instance, when integrated with BEVDet, the nuScenes Detection Score (NDS) rose from 48.8% to 54.8%.
Velocity Accuracy
The most notable improvement occurred in velocity estimation. By effectively denoising radar signals, the model leveraged radar's Doppler measurements more accurately. For the BEVDet baseline, the mean Velocity Error (mAVE) dropped from 0.950 to 0.463—a nearly 50% reduction.
Efficiency and Practicality
Despite the inclusion of diffusion modeling, the implementation is lightweight, adding negligible computational overhead. Furthermore, the "LiDAR-free" nature of the training and inference makes it highly practical for real-world deployment where LiDAR data may be unavailable.
Conclusion and Future Work
While the framework robustly handles radar sparsity and noise, it remains sensitive to sensor calibration. Misalignment between camera and radar can degrade the 3D-to-2D projection accuracy. Additionally, the model cannot synthesize data for objects that produce zero radar returns, falling back on camera-based depth estimation. Future research may explore more complex XYZ measurement denoising and the integration of temporal data to further enhance detection stability.
Summary updated