Radar Panoptic Transformer: Semantic-Guided Mask Queries for Dynamic Scene Understanding

Mayank Mayank, Bharanidhar Duraisamy, Rohit Mohan, Abhinav ValadaarXiv 2026 · arXiv

TL;DR

The Radar Panoptic Transformer (RPT) is an end-to-end, point-based architecture designed to solve panoptic segmentation for sparse automotive radar data. By integrating radar-specific features like Doppler velocity and Radar Cross Section (RCS) with a semantic-guided mask decoder, RPT achieves state-of-the-art results on the RadarScenes benchmark and operates at a real-time 31 Hz, significantly outperforming previous geometric clustering-based methods.

What Problem Does This Paper Address?

Panoptic segmentation—the task of assigning a semantic label (e.g., "car," "static") and a unique instance ID to every point in a scan—is a cornerstone of autonomous driving. While well-studied for cameras and LiDAR, radar remains a "frontier" due to its inherent difficulty.

Radar point clouds are notoriously sparse and non-uniformly sampled; a car 200 meters away might only return two or three detections. Furthermore, the data is dominated by static background noise, making it incredibly difficult to separate a few dynamic object points from a sea of static returns. Traditional methods often rely on geometric clustering (grouping points by distance), but this fails when objects are sparse or when noisy, high-Doppler static returns mimic moving targets.

Methods

RPT moves away from rigid clustering pipelines in favor of a learnable, transformer-based approach tailored for radar:

  1. Radar-Adaptive Spatio-Temporal Backbone: The network uses KPConv for local geometric encoding and Grouped Vector Attention (GVA) to integrate context from previous scans (pose-aligned). Crucially, it learns a per-point neighborhood radius conditioned on the range, azimuth, and RCS of the detection. This allows the network to adapt its receptive field to the varying density of radar data.
  2. Semantic-Guided Mask Decoder: Instead of using generic, randomly initialized queries, RPT uses its own semantic predictions to "seed" object queries at likely foreground locations. It then uses masked cross-attention, where the query and point-wise features interact to refine the instance mask without ever needing an explicit clustering step.
  3. RCS-Consistency Loss: The authors introduce a novel auxiliary loss that regularizes predicted instances based on their Radar Cross Section (RCS) statistics (mean and variance). This ensures that the predicted "object" physically resembles the reflectivity profile of the target class.

Datasets

The authors evaluated RPT on two primary benchmarks:

  • RadarScenes: A large-scale dataset featuring four 77 GHz radars with a 100m range and point-level semantic and instance annotations.
  • MAN TruckScenes: A more challenging, long-range dataset with six 4D radars providing 360° coverage up to 230m. This dataset is particularly difficult due to the extreme sparsity of detections at long distances.

Key Insights and Ideas

  • Semantics First, Instances Second: By using semantic probability to initialize queries, the model avoids wasting computation on the dominant static background. This "semantic guidance" is the key to finding the "needle in the haystack" of dynamic objects.
  • Dynamic Receptive Fields: The realization that fixed-radius neighborhoods are suboptimal for radar is vital. Learning the radius based on range and azimuth allows the model to aggregate more points for distant, sparse objects while maintaining high resolution for nearby ones.
  • Doppler as a Differentiator: The model leverages ego-motion compensated Doppler velocity as a core feature, allowing it to remain robust even when static returns exhibit spurious high-Doppler measurements (radar artifacts).
  • End-to-End Efficiency: Unlike previous multi-stage refinement pipelines, RPT is a single-stage network, which is why it can exceed the sensor's frame rate (31 Hz vs. 17-20 Hz).

Key Results

  • RadarScenes: RPT set a new state-of-the-art with 71.6% mIoU (semantic) and 82.2% PQ (panoptic quality), outperforming the previous best (SemRaFiner) by a healthy margin.
  • Long-Range Performance: On the MAN TruckScenes dataset, RPT outperformed strong baselines like Mask3D and PTv3, particularly in the "truck" category, where it showed an 11.8-point gain in PQ over Mask3D.
  • Qualitative Superiority: Qualitative results show RPT is much better at separating adjacent vehicles with similar motion profiles, whereas older methods often merged them into a single "blob."
  • Real-Time Inference: The model processes scans in ~32ms on an NVIDIA RTX A6000, making it viable for deployment on autonomous vehicle hardware.

Limitations and Future Work

  • VRU Data Scarcity: In the long-range TruckScenes dataset, the authors had to restrict evaluation to cars and trucks because pedestrians and two-wheelers were too sparsely represented to provide statistically significant results.
  • Singleton Instances: While RPT is better than clustering, instances represented by only a single radar return (singletons) remain a significant challenge for any panoptic architecture.
  • Future Directions: The authors suggest that integrating RPT into a multi-modal setup (e.g., Radar + Camera) could further improve the robustness of semantic seeding, especially for static or very slow-moving objects.

Summary updated

Abstract

Automotive radar provides long-range sensing and direct Doppler velocity measurements, but its sparse and non-uniformly sampled point clouds pose significant challenges for panoptic segmentation. Dynamic objects often yield only a few detections amid a dominant static background, providing limiting the geometric cues for separating object instances. We propose \emph{Radar Panoptic Transformer (RPT)}, an end-to-end point-based network that incorporates radar-adaptive feature extraction with semantic-guided mask prediction. Its backbone aggregates pose-aligned temporal context through grouped vector attention and learns per-point neighborhood radii conditioned on range, azimuth, and radar cross section (RCS). Point-wise semantic predictions initialize queries at likely foreground detections and guide masked cross-attention through query-point class agreement, producing instance masks without geometric clustering. An auxiliary RCS-consistency loss matches instance-level statistics between predicted and annotated instances. RPT achieves state-of-the-art performance on RadarScenes with $71.6\%$ mIoU and $82.2\%$ PQ. We further evaluate RPT on MAN TruckScenes, where it outperforms the evaluated baselines in a long-range, full-surround radar setting with sensing ranges up to $230\,\mathrm{m}$. RPT achieves real-time end-to-end inference at $31$\,Hz on RadarScenes, exceeding the sensor frame rate.
BibTeX
@misc{mayank2026radar,
  author = {Mayank Mayank and Bharanidhar Duraisamy and Rohit Mohan and Abhinav Valada},
  title = {{Radar Panoptic Transformer: Semantic-Guided Mask Queries for Dynamic Scene Understanding}},
  year = {2026},
  eprint = {2611.20203},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.20203},
}
Figure