TL;DR
SIFormer (Scene-Instance aware transFormer) is a novel multi-modal 3D object detection framework that bridges the gap between BEV-level and perspective-level radar-camera fusion. By injecting 2D perspective instance cues to guide Bird's-Eye-View (BEV) feature activation, it overcomes the weak geometry and sparsity inherent in 4D radar data, achieving state-of-the-art performance across multiple autonomous driving benchmarks.
What Problem Does This Paper Address?
Reliable 3D object detection is a cornerstone of autonomous driving. While LiDAR provides excellent 3D geometry, its high cost and vulnerability to adverse weather conditions limit its widespread adoption. 4D millimeter-wave radar has emerged as an affordable, weather-resilient alternative that adds elevation and velocity data to traditional radar capabilities. However, 4D radar data remains highly sparse and noisy.
To compensate, researchers fuse radar data with camera images. Existing fusion methods generally fall into two categories, each with a distinct trade-off:
- BEV-Level Fusion: Camera features are projected into a 3D/BEV volume and combined with radar features. This provides excellent global scene context but treats foreground and background features equally, leading to blurred object features and weak instance focus.
- Perspective-Level Fusion: 2D image proposals are generated first and then refined using radar data. This captures fine-grained instance details but lacks holistic scene context and suffers from errors cascading from the initial 2D proposal network.
As shown in the paper's visualization comparison, 4D radar provides much weaker geometric cues than dense LiDAR, making instance mining in BEV space exceptionally challenging.
SIFormer addresses this dilemma by introducing a cross-view correlation mechanism that combines the strengths of both paradigms, ensuring both global scene understanding and sharp instance awareness.
Methods
SIFormer consists of three primary novel components integrated into a transformer-based architecture:
- Sparse Scene Integration (SSI): To prevent background noise and geometric inaccuracies from polluting the BEV space during view transformation, SSI applies two filtering mechanisms. Segmentation-Guided Weighting (SGW) uses a lightweight network to predict foreground regions and re-weight context features. Depth-Guided Weighting (DGW) retains only the top 25% of discrete depth probabilities, preventing camera features from being erroneously projected into low-probability regions.
- Cross-View Correlation (CVC): This module acts as the bridge between views. It takes 2D object proposals from the image (using a standard detector like Cascade Mask R-CNN) and uses a learnable token to map these local perspective instance cues directly into the global BEV feature map. This explicitly activates instance-relevant regions in the scene feature map.
- Instance Enhance Attention (IEA): Once instance regions are activated, they serve as queries in a transformer block. The IEA uses a Semantic Enhancement Module (SEM) to aggregate multi-modal semantic features via 3D deformable cross-attention, and a Geometry Enhancement Module (GEM) to efficiently fuse radar geometric features using a U-Net architecture and neighborhood cross-attention.
Datasets
The authors evaluated SIFormer on three major autonomous driving datasets:
- View-of-Delft (VoD): A multi-class road user dataset containing 3+1D radar, LiDAR, and camera data captured in urban environments.
- TJ4DRadSet: A 4D radar dataset featuring challenging real-world driving scenarios such as nighttime, severe glare, under-bridge scenes, and defocused images.
- nuScenes: A large-scale benchmark used to demonstrate SIFormer's adaptability to conventional 3D radar systems.
Key Insights and Ideas
- Leveraging 2D Clarity for 3D Sparsity: Because 4D radar geometry is too weak to reliably isolate instances directly in BEV space, the clear object boundaries available in 2D image space serve as an ideal anchor. Mapping 2D instance cues into BEV space effectively "wakes up" the network to where objects actually reside.
- Filtering Before Fusing: Treating all spatial features equally during view transformation dilutes feature contrast. By aggressively filtering out the bottom 75% of depth bins and background semantics before generating the final BEV representation, the model significantly reduces background interference.
- Graceful Degradation & Robustness: The cross-view design creates a mutually reinforcing system. If the camera fails or suffers from poor lighting, the radar's geometric context keeps the system stable; if the radar returns noisy multi-path reflections, the perspective-level cues prevent false positives.
Key Results
SIFormer achieved state-of-the-art performance across all tested benchmarks:
- View-of-Delft: With LiDAR depth supervision during training, SIFormer achieved 63.32% mAP in the Entire Annotated Area and 83.06% mAP in the Driving Corridor, outperforming strong baselines like LXL and SGDet3D. Even without LiDAR supervision, it remained highly competitive.
- TJ4DRadSet: SIFormer achieved 46.51% mAP for 3D detection ($AP_{3D}$) and 50.42% mAP for BEV detection ($AP_{BEV}$), outperforming previous radar-camera fusion models by a large margin.
- Calibration Robustness: In tests simulating sensor misalignment (up to $\pm20^\circ$ rotational and $\pm1.5\text{m}$ translational disturbance), SIFormer maintained a significantly higher mAP (37.45%) than the baseline LXL (31.45%), proving its readiness for imperfect real-world sensor setups.
Limitations and Future Work
Despite its strong detection capabilities, the authors note two primary limitations:
- Inference Speed: Running at 6.9 FPS, the model is currently too slow for real-time deployment on standard automotive hardware. The authors plan to investigate lightweight variants using network pruning and knowledge distillation.
- Lack of Temporal Modeling: The current framework operates on single frames (or simple accumulated static scans). Incorporating explicit temporal modeling across frames is highlighted as a crucial next step to improve tracking and further smooth out radar noise.
Summary updated