Full-Velocity Radar Returns by Radar-Camera Fusion

Yunfei Long, Daniel Morris, Xiaoming Liu, Marcos Castro, Punarjay Chakravarty, Praveen Narayanan

TL;DR

This paper introduces a novel method to "upgrade" sparse Doppler radar data by fusing it with camera imagery. By combining radial velocity from radar with optical flow from a camera, the authors derive a closed-form solution to estimate an object's full 2D/3D velocity, significantly improving motion compensation and radar point cloud accumulation for autonomous driving tasks.

What Problem Does This Paper Address?

Doppler radar is a staple in autonomous vehicle sensor suites because it can directly measure how fast an object is moving toward or away from the sensor (radial velocity). However, it has a major blind spot: it cannot measure motion perpendicular to the sensor (tangential velocity). An object moving perfectly sideways relative to a radar sensor will appear to have zero velocity.

This "missing" component makes it difficult to accurately estimate the true speed of surrounding vehicles. Furthermore, because radar data is extremely sparse compared to LiDAR, researchers often try to "accumulate" multiple frames of radar data to create a denser point cloud. Without knowing the full velocity of moving objects, these accumulated points become smeared or misaligned, rendering them useless for downstream tasks like object detection or pose estimation.

Methods

The authors propose a two-stage approach to solve for full velocity:

  1. Closed-Form Geometric Fusion: The core contribution is a mathematical derivation that combines two constraints. First, the radar provides a radial velocity component. Second, the camera provides "optical flow"—the 2D motion of pixels between frames. Geometrically, the optical flow defines a set of possible 3D motions that project to the same image displacement, while the radial velocity defines another set. The intersection of these two constraints yields a unique, closed-form solution for the full 3D velocity vector.
Figure 1: (a) Full motion cannot be determined with a single sen- sor: all motions ending on the blue dashed line (i.e., blue dashed arrows) map to the same optical flow and all motions terminated on the red dashed line (i.e., red dashed arrows) fit the same ra- dial motion. However, with a radar-camera pair, the full motion can be uniquely decided: only the motion drawn in black satis- fies both optical flow and radial motion. (b) Optical flow in the camera-image and (c) a bird’s-eye view of the observed vehicle. This shows measured radar points with radial velocity (red), our predicted point-wise, full velocity (black), and ground truth full velocity of the vehicle (green).
  1. Radar-2-Pixel (R2P) Association: A major practical challenge is that radar points are low-resolution and don't always project perfectly onto the corresponding pixels in an image due to sensor noise or beam-width issues. The authors developed a neural network (R2P) that looks at a neighborhood around the raw projection and predicts the probability that a specific pixel is the "true" match for a radar return. This network also helps identify when a radar point is occluded in the camera view.
The R2P pipeline: Using a neural network to find the best pixel-to-radar association for velocity estimation.

Datasets

The researchers evaluated their method using the nuScenes dataset, a large-scale public dataset for autonomous driving that includes synchronized radar, camera, and LiDAR data. They used a subset of thousands of samples for training and testing, specifically focusing on moving objects where velocity estimation is most critical.

Key Insights and Ideas

  • Complementary Constraints: The "aha moment" is realizing that the camera's weakness (depth/scale ambiguity in motion) is exactly what the radar's radial measurement resolves, and vice versa.
  • Upgrading vs. Regressing: Unlike previous deep learning approaches that try to "learn" full velocity from data, this method uses a deterministic physical model. The neural network is only used for the association task (matching pixels to points), while the physics is handled by the closed-form equation.
  • Enabling Accumulation: By accurately estimating point-wise velocity, the authors can "re-project" old radar points to their current positions. This transforms a sparse, noisy radar sweep into a dense, coherent point cloud that looks much more like a LiDAR scan.

Key Results

The proposed method showed dramatic improvements over baseline methods that rely only on radial velocity:

  • Velocity Accuracy: The error in tangential velocity estimation was reduced to just 21% of the baseline error.
  • Object-Wise Estimation: When compared to CenterFusion (a state-of-the-art fusion model), this method reduced object velocity error from 0.826 m/s to 0.451 m/s.
  • Downstream Impact: Using the "full-velocity" accumulated point clouds improved 3D object pose estimation (Intersection over Union) from 0.462 to 0.546.
Qualitative results showing how full-velocity compensation (e) creates a much tighter, more accurate point cloud around a vehicle compared to no compensation (c) or radial-only compensation (d).

Limitations and Future Work

The primary limitation is occlusion. If an object is visible to the radar but hidden from the camera (e.g., behind a thin pillar or in very low light where optical flow fails), the fusion cannot occur. The authors also note that the method currently assumes objects are rigid; non-rigid motion (like a pedestrian's swinging arms) might introduce errors.

Future work could involve extending this framework to Doppler LiDAR or exploring more sophisticated ways to integrate these point-wise velocities directly into end-to-end 3D detection networks.

Summary updated

Abstract

A distinctive feature of Doppler radar is the measurement of velocity in the radial direction for radar points. However, the missing tangential velocity component hampers object velocity estimation as well as temporal integration of radar sweeps in dynamic scenes. Recognizing that fusing camera with radar provides complementary information to radar, in this paper we present a closed-form solution for the point-wise, full-velocity estimate of Doppler returns using the corresponding optical flow from camera images. Additionally, we address the association problem between radar returns and camera images with a neural network that is trained to estimate radar-camera correspondences. Experimental results on the nuScenes dataset verify the validity of the method and show significant improvements over the state-of-the-art in velocity estimation and accumulation of radar points.
BibTeX
@inproceedings{long2021full,
  author = {Yunfei Long and Daniel Morris and Xiaoming Liu and Marcos Castro and Punarjay Chakravarty and Praveen Narayanan},
  title = {{Full-Velocity Radar Returns by Radar-Camera Fusion}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2021},
  doi = {10.1109/iccv48922.2021.01589},
}
Figure