Body-Centred Metric Volumes for Pose Estimation from Raw 4D Radar

Millimetre-wave radar can estimate a person's 3D skeleton in the dark, through smoke, and without recording a recognisable image. That makes it attractive for bedrooms, hospital wards and care homes. The most accurate radar pose systems skip the usual step of reducing radar echoes to a sparse point list. Instead they read the raw 4D radar tensor, a dense grid of reflected energy indexed by range, azimuth angle, elevation angle and Doppler velocity, and they treat it with 3D convolutions borrowed from volumetric medical imaging. Two problems come with this. The tensor is huge, up to 100 MB per frame in the RT-Pose benchmark. It is also not a Cartesian volume: an angular bin covers more space the farther it is from the sensor, so a fixed 3D kernel sees a different amount of body at 2 m than at 8 m.

We propose a two-stage fix. A cheap coarse pass first finds where the person is. The system then resamples only a small, body-sized cube around that location from the radar's native spherical grid into a fixed metric grid, for example a 2 m cube at a few centimetres per voxel, aligned to the body centre. Every person then looks geometrically the same to the network regardless of distance. Each frame's input is also small enough to stack many frames into a long temporal window. The claim is that this crop-and-resample step, not a larger backbone, is what removes two failure modes RT-Pose reports: accuracy that depends on range, and degradation on complex, fast-changing actions.

Core Claim

The closest existing system, HRRadarPose in RT-Pose: A 4D Radar-Tensor based 3D Human Pose Estimation and Localization Benchmark, runs ordinary 3D convolutions directly over the spherical radar tensor. It reaches 9.93 cm mean per-joint position error (MPJPE, the average distance between predicted and true joints), against 18.43 cm for RF-Pose3D. Its authors report that it degrades on complex actions. The geometric distortion is simple to state. If an angular bin has width $\Delta\theta$ radians, then at range $r$ it spans roughly $r\,\Delta\theta$ metres. A body part of size $h$ therefore covers about $h/(r\,\Delta\theta)$ bins, which is four times fewer bins at 8 m than at 2 m across RT-Pose's capture envelope. A convolution with fixed bin-sized kernels must learn a separate filter for every distance.

We claim that converting a body-centred crop to a fixed metric grid makes pose error nearly independent of range. We also claim that the memory freed by cropping, spent on temporal depth, reduces error on RT-Pose's hardest action-complexity levels more than the same compute spent on a wider backbone.

The design is assembled from specific prior components. RadarOcc: Robust 3D Occupancy Prediction with 4D Imaging Radar motivates respecting spherical geometry. It keeps features in spherical coordinates with sparse 3D convolutions and reads them into a Cartesian scene grid with learned deformable queries. However, it targets whole driving scenes at 3.3 FPS, works on single frames, and never regresses articulated joints. SegVol: Universal and Interactive Volumetric Medical Image Segmentation supplies the coarse-then-local idea. Its zoom-out-zoom-in inference localises a region cheaply and then refines it at high resolution, about 17× faster than sliding windows on CT. We carry that idea into a setting where the "zoom-in" also changes coordinate system.

The proposal differs from RadarOcc in three ways. The Cartesian grid is per-person and moves with the body. The resampling is explicit rather than learned attention. The saved budget goes to time rather than scene extent. It differs from SegVol in that the data are sparse, noisy radar returns rather than dense tissue contrast.

Research Program

Stage one is a coarse localiser. It runs on a downsampled, Doppler-summed tensor and predicts a body-centre confidence map, much like HRRadarPose's own centre head. Its only job is to reach high recall within a margin smaller than the crop half-width.

Stage two resamples the native tensor inside the body-centred cube into a metric grid. It uses trilinear interpolation of per-bin magnitudes and compact Doppler descriptors, in the style of RadarOcc's Doppler bin descriptors. Upsampling at far range does not create information; it only standardises geometry. To tell the network how blurry each voxel really is, we add one channel encoding the native bin footprint $r\,\Delta\theta$ at that voxel.

Stage three stacks $T$ consecutive crops and processes them with a factorised spatio-temporal volumetric network, in the spirit of Parallel Separable 3D Convolution for Video and Volumetric Data Understanding. A 3D U-Net-style spatial encoder, following 3D U-Net: Learning Dense Volumetric Segmentation from Sparse Annotation, is followed by temporal mixing. The network regresses joint offsets relative to the crop centre with a bone-length loss. We will sweep $T$ from 1 to 32 at a fixed memory budget.

The analysis has two parts. First, we measure error as a function of range and of RT-Pose's six action-complexity levels. Second, we check whether the gain on complex actions tracks $T$, which would mean temporal context is the active ingredient, or appears even at $T=1$, which would mean geometry is the active ingredient.

The privacy argument is secondary but real. The stage-two input contains only the person's immediate surroundings, not the room. We will state this as a design property, not measure it as a privacy guarantee.

Baselines

The naive reference point places a training-set mean pose at the stage-one body centre. This shows how much of any MPJPE figure comes from localisation alone.

The strongest existing method is HRRadarPose from RT-Pose, which reports 9.93 cm MPJPE on RT-Pose. It is the right competitor because it already consumes the raw tensor with volumetric convolutions; our proposal changes only where and in what coordinates those convolutions run. We will retrain it with the released protocol and splits. If our reproduction differs materially from 9.93 cm, we will compare against our reproduction and report both. Beating it means lower overall MPJPE at equal or lower training GPU-hours and peak memory. It also means a clearly flatter error-versus-range curve and lower error on the top two complexity levels, each with confidence intervals over three seeds.

The second strong competitor is a RadarOcc-style pipeline adapted to pose. It uses top-$N$ per-range-bin sparsification, spherical sparse convolutions, and deformable Cartesian queries followed by our pose head. RadarOcc only reports occupancy results on K-Radar, where it improves mean IoU by 40–57% over point-cloud methods. There is no reported result found on Lacuna for a RadarOcc-style pose model on RT-Pose, so we will build and tune it ourselves.

The CFAR point-cloud route is represented by Diffusion Model Is a Good Pose Estimator from 3D RF-Vision. This is mmDiff, a conditional diffusion model over radar points, which improves MPJPE by 12.8% over P4Transformer on mmBody. There is no reported result found on Lacuna for mmDiff on RT-Pose, so we will run it on CFAR points extracted from the same tensors.

All learned baselines get the same training data, the same number of hyperparameter trials, and a matched GPU-hour budget.

Three ablations isolate the key ingredient. The first crops in spherical coordinates without resampling. The second resamples the whole scene to a coarse Cartesian grid without cropping. The third removes the footprint channel.

Why not just…?

Why not just resample the entire tensor to a Cartesian grid once and run HRRadarPose on it? This removes the distortion but not the memory problem. Covering 2–8 m at centimetre resolution multiplies voxel count far beyond the native tensor, which forces either coarse voxels or short windows. The whole-scene coarse-resample ablation tests this directly at matched memory. If it matches our method, the contribution reduces to "resample," and we will say so.

Why not just use RadarOcc's sparsification and learned spherical-to-Cartesian queries? This is the real threat. Learned deformable queries could absorb range distortion without explicit resampling, and top-$N$ sparsification already cuts memory. Our bet is that per-person explicit resampling makes range invariance hold by construction, which matters on a dataset of only 240 sequences. The adapted RadarOcc baseline, evaluated on the range-stratified error curve, settles whether the explicit route is needed. It also tests whether that route is cheaper, given RadarOcc's 3.3 FPS on driving scenes.

Why not just smooth HRRadarPose outputs over time? Post-hoc filtering cannot restore information the per-frame network never extracted. Doppler Prompting for Stable mmWave-based Human Pose Estimation reports that earlier models sit at 13–15 mm/frame velocity error, while gating Doppler before spatial reasoning reaches 9.78 mm/frame on HuPR. We will nonetheless run a tuned temporal filter on HRRadarPose outputs and compare it with our $T$ sweep on the complex-action levels. That comparison checks whether learned temporal volumes outperform cheap smoothing.

Proposed Experiments

The main benchmark is RT-Pose: 240 sequences, 72,000 frames, raw tensors with LiDAR-derived 3D ground truth, six complexity levels, and indoor, outdoor, rain, low-light and occlusion conditions. We report MPJPE, per-joint error with emphasis on wrists and ankles, and velocity error for jitter. We also report peak memory, throughput and latency.

There are three decisive checks. The first is range sensitivity. We plot MPJPE in 1 m range bins from 2 to 8 m and fit a slope for each method. The prediction is a near-zero slope for our model and the no-resample ablation, and a positive slope for HRRadarPose and the spherical-crop ablation. If HRRadarPose's slope is already flat, the geometric motivation is wrong and we will report that. The second is the complexity-by-temporal-depth interaction: error on levels 5–6 should fall with $T$ markedly more than on levels 1–2. The third is stage-one sensitivity. We inject controlled centre errors to find the crop margin at which accuracy collapses, and report the localiser's recall under rain and occlusion.

As a secondary check, we will apply the method to the dual-radar recordings of HuPR: A Benchmark for Human Pose Estimation Using Millimeter Wave Radar. There, both radars can be resampled into the same body-centred metric cube instead of being fused by cross-attention. HuPR's reported AP is 63.4. This comparison depends on recovering the radars' relative calibration and uses 2D keypoint ground truth, so we treat it as supporting evidence only.