GET-UP: GEomeTric-Aware Depth Estimation with Radar Points UPsampling

Huawei Sun, Zixu Wang, Hao Feng, Julius Ott, Lorenzo Servadei, Robert WilleWACV 2025

TL;DR

This paper introduces GET-UP, a novel radar-camera depth estimation method that leverages geometric information from sparse radar point clouds. By upsampling radar points and integrating 2D and 3D features using an attention-enhanced GNN, GET-UP achieves state-of-the-art performance on the nuScenes dataset, significantly outperforming previous methods.

What Problem Does This Paper Address?

Accurate 3D scene understanding is crucial for autonomous driving, with depth estimation being a key component. While camera-based methods have advanced, they struggle in adverse weather. Radar offers robustness but suffers from sparsity and noise, making direct projection onto image planes problematic for depth estimation. Existing radar-camera approaches often fail to fully utilize the geometric richness of radar point clouds, leading to suboptimal performance. This paper addresses the challenge of effectively integrating sparse and noisy radar data with camera information for accurate depth estimation, specifically by better exploiting the 3D geometric properties of radar.

Methods

GET-UP employs a multi-stage approach:

  • 2D Feature Extraction: An Adaptive Sparse Convolution Block (ASCB) is introduced to handle sparse radar projections. This block adaptively adjusts convolution kernel sizes based on radar point depth, categorizing detections into distance groups to improve feature propagation. This is followed by a ResNet backbone.
  • 3D Feature Extraction: An attention-enhanced Dynamic Graph Convolutional Neural Network (DGCNN) is used to extract 3D geometry-aware features from the radar point cloud. This model incorporates 2D features via cross-attention mechanisms.
  • 2D-3D Feature Aggregation: Features from the 2D and 3D extraction submodules are fused using cross-attention to create more robust representations.
  • Point Cloud Upsampling: A novel point cloud upsampling submodule is integrated. This module uses LiDAR data to generate ground truth for upsampling radar points, aiming to densify the point cloud and rectify point positions. This is crucial for improving the spatial accuracy of radar features.
  • Decoder: A gated fusion mechanism combines the refined radar features with image features, which are then fed into a decoder (based on the BTS model) for final depth map estimation.

Datasets

The paper uses the nuScenes dataset for training and evaluation.

Key Insights and Ideas

  • Geometric Information is Key: The paper strongly emphasizes that simply projecting radar points onto an image plane and extracting 2D features is insufficient. Exploiting the inherent 3D geometric structure of radar point clouds is critical for improved depth estimation.
  • Upsampling as a Solution to Sparsity: Instead of just densifying radar projections in 2D, the paper proposes upsampling the 3D radar point cloud itself. This directly addresses the sparsity issue in the 3D domain and leverages LiDAR for accurate ground truth.
  • Attention for Feature Fusion: The use of attention mechanisms in both the 3D feature extraction and 2D-3D aggregation stages allows for effective integration of information from different modalities and scales.
  • Adaptive Convolutions for Radar: The ASCB's ability to adapt kernel sizes based on radar point depth is a clever way to handle the varying density and scale of radar detections across different distances.

Key Results

GET-UP achieves state-of-the-art performance on the nuScenes dataset.

  • It improves upon the previous best-performing model by 15.3% in MAE and 14.7% in RMSE.
  • It significantly outperforms image-only methods, showing a 33.8% improvement in MAE and 23.3% in RMSE over the BTS baseline.
  • Qualitatively, GET-UP produces depth maps with clearer object boundaries and greater robustness compared to baselines.

Limitations and Future Work

  • Limitations: The paper acknowledges that the performance of the upsampling module is sensitive to the choice of nu (number of upsampling units) and NL (number of upsampled points). The selection of NL is particularly important for ensuring efficacy.
  • Future Work: The authors suggest exploring diverse upsampling algorithms for radar point clouds and refining the integration of 2D and 3D radar features as promising directions for future research.

Summary updated

BibTeX
@inproceedings{sun2025get,
  author = {Huawei Sun and Zixu Wang and Hao Feng and Julius Ott and Lorenzo Servadei and Robert Wille},
  title = {{GET-UP: GEomeTric-Aware Depth Estimation with Radar Points UPsampling}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2025},
}
Figure