Abstract
Multimodal 3D object detection, integrating LiDAR and camera data, faces significant performance degradation in adverse weather conditions like snow, rain, and fog. These conditions introduce "phantom" noise points and cause signal attenuation, leading to sparse point clouds. We present EquiDetect, a framework that addresses these challenges through equirectangular point reconstruction and physics-aligned synthetic noise generation. By projecting LiDAR data into a range image and using 2D camera detections to "hallucinate" missing points, EquiDetect maintains high detection accuracy. Experimental results on KITTI, S-KITTI, Dense, and CADC datasets demonstrate that EquiDetect consistently outperforms state-of-the-art models in challenging environments.
Introduction
Autonomous vehicles rely heavily on multimodal 3D object detection for safe navigation. While current models perform exceptionally well in clear weather, adverse conditions remain a major hurdle. Weather particles interfere with LiDAR signals in two primary ways: creating noise through reflections off snowflakes or raindrops, and causing sparsity (holes) through signal absorption. Additionally, the lack of high-quality, labeled real-world data for extreme weather necessitates the use of simulators. However, existing simulators often fail to align synthetic LiDAR noise with camera image disturbances, leading to poor generalization in real-world deployments.
Methodology
EquiDetect introduces several key modules to mitigate weather-induced sensor degradation:
Equirectangular Point Reconstruction
The framework projects 3D LiDAR points into a 2D equirectangular range image. This transformation enables the application of distance-constrained denoising techniques, effectively filtering out weather-related "phantom" points as if they were image noise.
Object-Centric Ray Generator
To address signal attenuation, EquiDetect utilizes 2D detections from the camera to pinpoint regions where objects are expected but LiDAR data is missing. A U-Net-based generator then reconstructs these "holes," ensuring the point cloud is sufficiently dense for accurate 3D detection, particularly for distant objects.
Multimodal Weather Noise Matching
To improve synthetic training data, the authors developed a radial manipulation method. This aligns synthetic weather particles in images with corresponding LiDAR noise points along the laser's path. This physics-based alignment significantly improves the realism of synthetic datasets compared to independent sensor simulation.
2D Feature Fusion and Restoration
The model incorporates a 2D feature fusion block and an auxiliary image restoration network to enhance the robustness of the camera-based features used for reconstruction and detection.
Unsupervised Domain Adaptation (UDA)
To bridge the gap between synthetic training data and real-world testing data, EquiDetect employs iterative feature perturbations. This aligns fine-grained feature distributions across domains more effectively than standard UDA methods.
Key Insights
- Targeted Reconstruction: Analysis shows an optimal point density for detection at various distances. Simply increasing point count is not always beneficial; reconstruction must be targeted to be effective (see Figure 1c in the overview).
- Range Image Efficiency: Treating LiDAR data as a 2D range image allows for efficient denoising using standard image processing logic, such as median filtering.
- Physics-Based Alignment: Enforcing radial alignment between sensors increases noise matching accuracy from ~28% in standard simulators to over 90% (see Figure 1d in the overview), closely mimicking real-world conditions.
Results and Discussion
EquiDetect was evaluated against state-of-the-art (SOTA) models like VirConv-S. On the S-KITTI dataset, it achieved an AP of 82.85% in fog and showed a 5.6 percentage point improvement in snow. Real-world testing on the Dense dataset confirmed its superior generalization, outperforming VirConv-S by 1.5% to 2% without target-domain labels.
Despite its performance, EquiDetect's reliance on 2D detection means it could be vulnerable in "whiteout" conditions where cameras are blinded. Future work will focus on further reducing computational overhead (currently 122.9 ms per frame) to meet the sub-100ms requirement for real-time high-speed driving.
Conclusion
EquiDetect represents a significant step forward in robust 3D object detection for adverse weather. By combining equirectangular reconstruction with physics-aligned multimodal simulation, it bridges the gap between clear-weather performance and real-world reliability in snow, rain, and fog.
Summary updated