Boosting Instance Awareness via Cross-View Correlation with 4D Radar and Camera for 3D Object Detection

Xiaokai Bai, Lianqing Zheng, Si-Yuan Cao, Xiaohan Zhang, Zhe Wu, Beinan Yu, Fang Wang, Jie Bai, Hui-Liang ShenarXiv 2026 · arXiv

TL;DR

SIFormer (Scene-Instance aware transFormer) is a novel multi-modal 3D object detection framework that bridges the gap between BEV-level and perspective-level radar-camera fusion. By injecting 2D perspective instance cues to guide Bird's-Eye-View (BEV) feature activation, it overcomes the weak geometry and sparsity inherent in 4D radar data, achieving state-of-the-art performance across multiple autonomous driving benchmarks.

What Problem Does This Paper Address?

Reliable 3D object detection is a cornerstone of autonomous driving. While LiDAR provides excellent 3D geometry, its high cost and vulnerability to adverse weather conditions limit its widespread adoption. 4D millimeter-wave radar has emerged as an affordable, weather-resilient alternative that adds elevation and velocity data to traditional radar capabilities. However, 4D radar data remains highly sparse and noisy.

To compensate, researchers fuse radar data with camera images. Existing fusion methods generally fall into two categories, each with a distinct trade-off:

  1. BEV-Level Fusion: Camera features are projected into a 3D/BEV volume and combined with radar features. This provides excellent global scene context but treats foreground and background features equally, leading to blurred object features and weak instance focus.
  2. Perspective-Level Fusion: 2D image proposals are generated first and then refined using radar data. This captures fine-grained instance details but lacks holistic scene context and suffers from errors cascading from the initial 2D proposal network.
Comparison of traditional fusion pipelines vs. SIFormer's cross-view fusion approach.

As shown in the paper's visualization comparison, 4D radar provides much weaker geometric cues than dense LiDAR, making instance mining in BEV space exceptionally challenging.

Fig. 2. Visualization comparison of LiDAR and 4D radar on the VoD dataset: the first row shows LiDAR, and the second row shows 4D radar. The first column displays 3D ground truth boxes and the point cloud projection onto the foreground mask, with purple points indicating valid points providing object information. The second and third columns show the point cloud projection onto the perspective View and bird’s-eye view, respectively. Dense LiDAR provides strong geometry, while sparse 4D radar only provides weak geometry.

SIFormer addresses this dilemma by introducing a cross-view correlation mechanism that combines the strengths of both paradigms, ensuring both global scene understanding and sharp instance awareness.

Methods

SIFormer consists of three primary novel components integrated into a transformer-based architecture:

  • Sparse Scene Integration (SSI): To prevent background noise and geometric inaccuracies from polluting the BEV space during view transformation, SSI applies two filtering mechanisms. Segmentation-Guided Weighting (SGW) uses a lightweight network to predict foreground regions and re-weight context features. Depth-Guided Weighting (DGW) retains only the top 25% of discrete depth probabilities, preventing camera features from being erroneously projected into low-probability regions.
  • Cross-View Correlation (CVC): This module acts as the bridge between views. It takes 2D object proposals from the image (using a standard detector like Cascade Mask R-CNN) and uses a learnable token to map these local perspective instance cues directly into the global BEV feature map. This explicitly activates instance-relevant regions in the scene feature map.
  • Instance Enhance Attention (IEA): Once instance regions are activated, they serve as queries in a transformer block. The IEA uses a Semantic Enhancement Module (SEM) to aggregate multi-modal semantic features via 3D deformable cross-attention, and a Geometry Enhancement Module (GEM) to efficiently fuse radar geometric features using a U-Net architecture and neighborhood cross-attention.

Datasets

The authors evaluated SIFormer on three major autonomous driving datasets:

  • View-of-Delft (VoD): A multi-class road user dataset containing 3+1D radar, LiDAR, and camera data captured in urban environments.
  • TJ4DRadSet: A 4D radar dataset featuring challenging real-world driving scenarios such as nighttime, severe glare, under-bridge scenes, and defocused images.
  • nuScenes: A large-scale benchmark used to demonstrate SIFormer's adaptability to conventional 3D radar systems.

Key Insights and Ideas

  • Leveraging 2D Clarity for 3D Sparsity: Because 4D radar geometry is too weak to reliably isolate instances directly in BEV space, the clear object boundaries available in 2D image space serve as an ideal anchor. Mapping 2D instance cues into BEV space effectively "wakes up" the network to where objects actually reside.
  • Filtering Before Fusing: Treating all spatial features equally during view transformation dilutes feature contrast. By aggressively filtering out the bottom 75% of depth bins and background semantics before generating the final BEV representation, the model significantly reduces background interference.
  • Graceful Degradation & Robustness: The cross-view design creates a mutually reinforcing system. If the camera fails or suffers from poor lighting, the radar's geometric context keeps the system stable; if the radar returns noisy multi-path reflections, the perspective-level cues prevent false positives.

Key Results

SIFormer achieved state-of-the-art performance across all tested benchmarks:

  • View-of-Delft: With LiDAR depth supervision during training, SIFormer achieved 63.32% mAP in the Entire Annotated Area and 83.06% mAP in the Driving Corridor, outperforming strong baselines like LXL and SGDet3D. Even without LiDAR supervision, it remained highly competitive.
  • TJ4DRadSet: SIFormer achieved 46.51% mAP for 3D detection ($AP_{3D}$) and 50.42% mAP for BEV detection ($AP_{BEV}$), outperforming previous radar-camera fusion models by a large margin.
  • Calibration Robustness: In tests simulating sensor misalignment (up to $\pm20^\circ$ rotational and $\pm1.5\text{m}$ translational disturbance), SIFormer maintained a significantly higher mAP (37.45%) than the baseline LXL (31.45%), proving its readiness for imperfect real-world sensor setups.

Limitations and Future Work

Despite its strong detection capabilities, the authors note two primary limitations:

  1. Inference Speed: Running at 6.9 FPS, the model is currently too slow for real-time deployment on standard automotive hardware. The authors plan to investigate lightweight variants using network pruning and knowledge distillation.
  2. Lack of Temporal Modeling: The current framework operates on single frames (or simple accumulated static scans). Incorporating explicit temporal modeling across frames is highlighted as a crucial next step to improve tracking and further smooth out radar noise.

Summary updated

Abstract

4D millimeter-wave radar has emerged as a promising sensing modality for autonomous driving due to its robustness and affordability. However, its sparse and weak geometric cues make reliable instance activation difficult, limiting the effectiveness of existing radar-camera fusion paradigms. BEV-level fusion offers global scene understanding but suffers from weak instance focus, while perspective-level fusion captures instance details but lacks holistic context. To address these limitations, we propose SIFormer, a scene-instance aware transformer for 3D object detection using 4D radar and camera. SIFormer first suppresses background noise during view transformation through segmentation- and depth-guided localization. It then introduces a cross-view activation mechanism that injects 2D instance cues into BEV space, enabling reliable instance awareness under weak radar geometry. Finally, a transformer-based fusion module aggregates complementary image semantics and radar geometry for robust perception. As a result, with the aim of enhancing instance awareness, SIFormer bridges the gap between the two paradigms, combining their complementary strengths to address inherent sparse nature of radar and improve detection accuracy. Experiments demonstrate that SIFormer achieves state-of-the-art performance on View-of-Delft, TJ4DRadSet and NuScenes datasets. Source code is available at github.com/shawnnnkb/SIFormer.
BibTeX
@misc{bai2026boosting,
  author = {Xiaokai Bai and Lianqing Zheng and Si-Yuan Cao and Xiaohan Zhang and Zhe Wu and Beinan Yu and Fang Wang and Jie Bai and Hui-Liang Shen},
  title = {{Boosting Instance Awareness via Cross-View Correlation with 4D Radar and Camera for 3D Object Detection}},
  year = {2026},
  eprint = {2602.20632},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2602.20632},
}

Loading related content...

Figure