Availability-aware Sensor Fusion via Unified Canonical Space

Dong-Hee Paek, Seung-Hyun Kong

TL;DR

Availability-aware Sensor Fusion (ASF) introduces a robust framework for autonomous driving perception that gracefully handles sensor degradation and unexpected failures across camera, LiDAR, and 4D Radar suites. By projecting multi-modal features into a unified canonical space and applying localized patch-wise cross-attention, ASF achieves state-of-the-art 3D object detection performance while maintaining real-time computational efficiency.

What Problem Does This Paper Address?

Autonomous vehicles rely on a combination of sensors—cameras for color/texture, LiDAR for high-resolution 3D structure, and 4D Radar for weather resilience and velocity tracking—to safely perceive their surroundings. However, combining these modalities effectively remains a challenge:

  1. Deeply Coupled Fusion (DCF) methods assume all sensors are continuously available and functioning perfectly. If a sensor degrades due to bad weather (e.g., LiDAR blinded by snow) or fails entirely, DCF performance plummets.
  2. Sensor-wise Cross-Attention Fusion (SCF) methods can handle missing sensors but suffer from massive computational complexity and struggle to find a unified feature representation across radically different data structures (2D images vs. 3D point clouds vs. sparse 4D tensors).

This paper addresses the need for a sensor fusion method that is both computationally efficient and robust against sensor degradation or failure without requiring network retraining when a sensor drops out.

Methods

The authors introduce Availability-aware Sensor Fusion (ASF), which ensures sensor independence while achieving a unified feature representation through three core components:

  • Unified Canonical Projection (UCP): Instead of directly concatenating raw features, UCP projects the Bird's Eye View (BEV) feature maps from each sensor into a standardized, canonical space using a shared reference query. This eliminates feature representation inconsistencies across different modalities.
  • Cross-Attention across Sensors Along Patches (CASAP): Because the BEV feature maps are already spatially aligned, ASF avoids expensive global cross-attention. Instead, CASAP applies cross-attention only across the sensors within the same spatial patch. This slashes computational complexity from $O(N_q N_s N_p)$ to $O(N_q N_s)$, where $N_s$ is the number of sensors and $N_p$ is the number of patches.
  • Sensor Combination Loss (SCL): During training, this loss function explicitly optimizes the network across all 7 possible sensor combinations (e.g., Camera-only, LiDAR+Radar, Full Suite). This forces the model to learn how to output accurate detections even when some sensors are completely unavailable.
Figure 1: Comparison of sensor fusion methods: (a) DCF (e.g., 3D-LRF (Chae et al., 2024)), (b) SCF (e.g., CMT (Yan et al., 2023)), and (c) ASF. FV, BEV, Obj., GT, Lcls and Lreg stand for ‘front-view’, ‘bird’s eye-view’, ‘objects’, ‘ground truths’, ‘classification loss’, and ‘regression loss’, respectively. ‘Feature coupling’ refers to methods that combine features from multiple sensors to create new features. Optional components are in dashed lines; for example, (Vora et al., 2020) combines camera and LiDAR features without transforming the camera viewpoint, while (Yan et al., 2023) fuses features through a transformer decoder head (Carion et al., 2020) without explicit feature coupling. ASF does not apply feature coupling to ensure independence between sensors.

Datasets

The authors trained and evaluated their method on the K-Radar dataset (benchmarks v1.0 and v2.0). K-Radar is a large-scale autonomous driving dataset uniquely suited for this research because it contains synchronized Camera, LiDAR, and 4D Radar data captured across a wide variety of severe weather conditions, including rain, fog, overcast, sleet, and heavy snow.

Key Insights and Ideas

  • True Redundancy via Alignment: High-dimensional t-SNE visualizations reveal that before UCP, sensor features are scattered chaotically. UCP successfully aligns them to a common baseline, allowing the subsequent CASAP module to form cohesive clusters based on environmental context.

    Figure 2: Visualization of feature representation with t-SNE (van der Maaten and Hinton, 2008) at different stages of ASF for ‘Sedan’ class. Red, green, blue, and gray dots represent features from camera, LiDAR, 4D Radar, and fused features, respectively. Symbols in solid lines such as circle and triangle, square, and star indicate normal, sleet, and heavy snow conditions, respectively. (a) Initial output features from sensor-specific encoders show inconsistent distribution across sensors. (b) After unified canonical projection (UCP), features become better aligned to the fused feature. (c) After cross-attention across sensors along patches (CASAP), features from available sensors form cohesive clusters (in dashed symbols) based on weather conditions. Note that in adverse weather, camera features show larger deviation due to the degradation. Additional visualizations are in Appendix B.
  • Dynamic Attention Redistribution: In adverse weather conditions like heavy snow, the system automatically shifts its attention weights away from the blinded camera and LiDAR, redistributing reliance toward the weather-robust 4D Radar.

  • Efficiency via Spatial Locality: By exploiting the fact that BEV patches are already spatially aligned, ASF eliminates the need for computationally heavy positional embeddings and global cross-attention, unlocking massive speedups.

Key Results

  • State-of-the-Art Accuracy: On the K-Radar v1.0 benchmark, ASF achieved a massive 20.1% absolute improvement in 3D Average Precision ($AP_{3D}$) (73.6% vs. 53.5% SOTA) and a 9.7% improvement in $AP_{BEV}$ (87.2% vs. 77.5% SOTA) at IoU=0.5.
  • Resilience to Failure: When simulating a completely obscured or damaged camera ($C^*$), the model's performance dropped by only 1.7% $AP_{3D}$ because it dynamically adapted to rely on the remaining LiDAR and Radar inputs.
  • Real-Time Speed: ASF runs at 20.5 Hz for a LiDAR+Radar configuration and 13.5 Hz for the full Camera+LiDAR+Radar suite on a single RTX 3090 GPU, comfortably exceeding the standard 10 Hz real-time threshold while maintaining a compact memory footprint (1.5–1.6 GB VRAM).

Limitations and Future Work

The primary limitation identified is the performance of the camera network backbone. The t-SNE visualizations show that camera features remain more isolated in the feature space compared to the tightly integrated LiDAR and 4D Radar features, particularly in bad weather.

Additionally, due to K-Radar's hardware setup (a narrow 12cm stereo camera baseline), the authors had to rely on a monocular camera method, which inherently limits long-range depth estimation. The authors suggest that integrating more advanced camera backbones and exploring wider stereo baselines are promising directions for future work.

Summary updated

Abstract

Sensor fusion of camera, LiDAR, and 4-dimensional (4D) Radar has brought a significant performance improvement in autonomous driving. However, there still exist fundamental challenges: deeply coupled fusion methods assume continuous sensor availability, making them vulnerable to sensor degradation and failure, whereas sensor-wise cross-attention fusion methods struggle with computational cost and unified feature representation. This paper presents availability-aware sensor fusion (ASF), a novel method that employs unified canonical projection (UCP) to enable consistency in all sensor features for fusion and cross-attention across sensors along patches (CASAP) to enhance robustness of sensor fusion against sensor degradation and failure. As a result, the proposed ASF shows a superior object detection performance to the existing state-of-the-art fusion methods under various weather and sensor degradation (or failure) conditions. Extensive experiments on the K-Radar dataset demonstrate that ASF achieves improvements of 9.7% in AP BEV (87.2%) and 20.1% in AP 3D (73.6%) in object detection at IoU=0.5, while requiring a low computational cost. All codes are available at https://github.com/kaist-avelab/k-radar.
BibTeX
@inproceedings{paek2025availability,
  author = {Dong-Hee Paek and Seung-Hyun Kong},
  title = {{Availability-aware Sensor Fusion via Unified Canonical Space}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2025},
  url = {https://openreview.net/forum?id=foy0BWzOFL},
}
Figure