Bootstrapping Autonomous Radars with Self-Supervised Learning

Yiduo Hao, Sohrab Madani, Junfeng Guan, Mohammed Alloulah, Saurabh Gupta, Haitham Hassanieh

TL;DR

Radical is a self-supervised learning framework that unlocks the power of massive, unlabeled radar datasets for autonomous driving. By combining radar-to-radar and radar-to-vision contrastive learning, it trains highly accurate radar-only object detectors that outperform supervised baselines by 5.8% mAP and excel in adverse weather conditions like heavy fog.

What Problem Does This Paper Address?

While cameras and LiDAR dominate autonomous vehicle perception, they fail catastrophically in adverse weather like fog, snowstorms, and heavy rain. Millimeter-wave (mmWave) radar can easily see through these conditions, making it a crucial safety redundancy.

However, training deep learning models for radar is severely bottlenecked by data annotation. Radar heatmaps look like blurry, uninterpretable blobs to humans, lacking sharp boundaries or recognizable shapes. Furthermore, radar signals exhibit "specularity" (mirror-like reflections), meaning only certain parts of a car reflect signals back to the receiver. This makes it incredibly difficult even for experts to draw precise bounding boxes. Consequently, only a tiny fraction (often around 10%) of collected radar data actually gets labeled, leaving vast amounts of raw data completely unused.

Figure 1. Millimeter wave radar heatmaps are uninterpretable to humans and are hence difficult to annotate.

Methods

The authors introduce Radical, a self-supervised pre-training framework that uses a joint embedding architecture to distill knowledge from a frozen vision model into a radar model without requiring explicit human labels.

Figure 2. Overall network of Radical. Knowledge is distilled from a pretrained vision model into a radar model. A mini-batch of B radar-vision pairs flow through network, whose encodings interact locally within the radar branch and globally across the radar and vision branches. That is, Radical is trained using a composite contrastive loss with intra- and cross-modal terms.

Radical's training pipeline relies on two core innovations:

  1. Composite Contrastive Loss: The framework optimizes two objectives simultaneously. A cross-modal loss aligns radar embeddings with paired camera images (using a pre-trained CLIP vision encoder), allowing the radar to inherit rich semantic priors like vehicle shapes and orientations. An intra-modal loss enforces consistency between different augmented views of the same radar data, helping the network learn radar-specific structural properties like sparsity.
  2. Radar MIMO Mask (RMM): Standard vision augmentations (like random cropping or color jitter) do not translate to radio frequency data. The authors propose RMM, a domain-specific augmentation tailored for Multiple-Input Multiple-Output (MIMO) radars. RMM manipulates the raw 3D complex signal tensor before it is collapsed into a 2D heatmap by applying:
  3. Antenna Dropout: Randomly masking out virtual antenna elements to simulate sensor degradation or obstructions.
  4. Random Phase Noise: Randomizing signal phases to mimic Doppler phase distortions caused by vehicle motion.

Additionally, standard vision augmentations like rotation and center cropping are adapted to operate natively within the radar's polar coordinate system.

Figure 3. Radar-specific augmentations. (a) Scene. (b) Original radar heatmap. (c) Zoomed-in region of cars. (d) Random Phase. (e) Antenna Dropout. (f) Rotation (Polar). (g) Center Cropping (Polar).

Datasets

The framework was evaluated using the Radatron dataset, which is uniquely suited for this work because it provides the raw multi-channel radar data necessary to implement the RMM augmentation. The data split consisted of:

  • 32K frames of unlabeled radar-vision pairs for self-supervised pre-training.
  • 13K annotated frames for supervised downstream fine-tuning.
  • 3K annotated frames for testing.

Key Insights and Ideas

  • Vision as a Teacher, Not a Crutch: Instead of using vision to generate noisy hard labels for radar training (which causes depth errors due to perspective projection), Radical uses cross-modal contrastive learning at the mutual information level. This allows the radar network to learn semantic concepts from vision during pre-training while remaining entirely vision-free at deployment time.
  • Naïve Augmentations Fail on Radar: Standard computer vision augmentations like vertical flipping or pixel thresholding actually hurt radar performance. Radar data is inherently asymmetric along its range axis, and because it is already extremely sparse, thresholding destroys critical signal data. Augmentations must respect the underlying physics of radar arrays.
  • Massive Label Efficiency: Pre-training with Radical makes downstream models incredibly data-efficient. When fine-tuned on just 1% of the available labeled data, Radical achieves an 11.5% mAP boost over a purely supervised model trained on the same tiny subset.

Key Results

The authors evaluated Radical on the challenging task of 2D bounding box car detection from standalone radar heatmaps:

  • Supervised Fine-Tuning: Fine-tuning the Radical pre-trained backbone yielded a 62.3% mAP, outperforming the fully supervised Radatron baseline (56.5% mAP) by 5.8%.
  • Frozen Backbone Evaluation: To test feature quality, the authors froze the pre-trained backbones and only trained the downstream detection heads. Radical achieved 52.6% mAP, decimating the randomly initialized baseline which scored just 22.1% mAP.
  • Adverse Weather Performance: In controlled fog experiments, Radical successfully resolved orientation and bounding box errors where standard supervised models failed due to radar smearing and specularity.

Limitations and Future Work

The authors acknowledge that Radical's cross-modal pre-training phase still relies on having co-located, paired camera and radar data, even if it doesn't require manual labels. If a dataset lacks paired visual data entirely, the cross-modal distillation cannot occur.

For future work, the authors suggest that Radical's self-supervised embeddings could be integrated with more advanced backbone and detector architectures, such as vision transformers or DETR-based models, to further push the limits of radar-only perception.

Summary updated

Abstract

The perception of autonomous vehicles using radars has attracted increased research interest due its ability to operate in fog and bad weather. However, training radar models is hindered by the cost and difficulty of annotating large-scale radar data. To overcome this bottleneck, we propose a self-supervised learning framework to leverage the large amount of unlabeled radar data to pre-train radar-only embeddings for self-driving perception tasks. The proposed method combines radar-to-radar and radar-to-vision contrastive losses to learn a general representation from unlabeled radar heatmaps paired with their corresponding camera images. When used for downstream object detection, we demonstrate that the proposed self-supervision framework can improve the accuracy of state-of-the-art supervised baselines by $5.8\%$ in mAP. Code is available at \url{https://github.com/yiduohao/Radical}.
BibTeX
@inproceedings{hao2024bootstrapping,
  author = {Yiduo Hao and Sohrab Madani and Junfeng Guan and Mohammed Alloulah and Saurabh Gupta and Haitham Hassanieh},
  title = {{Bootstrapping Autonomous Radars with Self-Supervised Learning}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2024},
  doi = {10.1109/cvpr52733.2024.01422},
}
Figure