Radar Supplies the Depth, Fog Sets the Weights: Physics-Weighted Camera–Radar Fusion of Multi-Person Motion in Rain

Cameras and radars fail in opposite places. In heavy rain or fog, a camera loses people at a distance, because the "veil" of light scattered by droplets washes out contrast more the farther away a person stands. A millimeter-wave radar sees through that veil. It loses people who stand close to each other instead, because their radio reflections add together at the receiver and blur into one. We propose to fuse the two sensors for multi-person 3D pose tracking using weights that come from these two physical facts. The camera's trust for each person drops with that person's range under an estimated fog density. The radar's trust drops with how close that person is to others in range, angle, and Doppler velocity. The fusion takes place on each person's motion trajectory written as a short sum of cosine waves, so the weights can differ between slow whole-body movement and fast limb detail.

The cross-link that makes this more than ordinary sensor fusion is that each sensor fixes the other's weak point. Optical models of the veil need per-pixel scene depth. Monocular depth estimators are known to break down under exactly the low contrast the veil creates. Radar measures each person's range directly, so it can supply the depth that the veil model needs. In the other direction, the camera resolves people who are adjacent in angle when they are near enough to see clearly, which is exactly when radar reflections from different people collide.

Core Claim

The claim is that, for multi-person 3D pose under rain or fog, the per-person, per-frequency reliability of each sensor can be predicted from two low-dimensional physical quantities. The first is the optical transmission $t_m = e^{-\beta d_m}$, the fraction of person $m$'s light that survives the haze. Here $\beta$ is the fog density and $d_m$ is the radar-measured range. The second is a radar separation score $s_m$, which is person $m$'s distance to the nearest other subject in range, angle, and Doppler, divided by the radar's resolution in each axis.

We claim that fusing trajectories with weights conditioned only on $(t_m, s_m, k)$, where $k$ indexes the temporal frequency band, will generalize to fog densities and crowd spacings never seen in training. We expect it to do so better than an end-to-end fusion network that has to learn reliability implicitly. Concretely, each joint trajectory over a window of $T$ frames is mapped to discrete cosine transform (DCT) coefficients $c^{(k)}$, $k = 0,\dots,K$, and fused by inverse-variance weighting:

$$\hat c^{(k)}_{m} = \frac{c^{(k)}_{m,\text{cam}} / \sigma^2_{\text{cam}}(t_m, k) + c^{(k)}_{m,\text{rad}} / \sigma^2_{\text{rad}}(s_m, k)}{1/\sigma^2_{\text{cam}}(t_m, k) + 1/\sigma^2_{\text{rad}}(s_m, k)}$$

In this equation, $\sigma^2_{\text{cam}}$ and $\sigma^2_{\text{rad}}$ are noise variances predicted by two tiny monotone networks that see only their physical inputs and the band index.

The frequency split matters because we expect the two failure modes to have different spectra. Radar sees bulk position and velocity well, since Doppler measures velocity directly. Its fine limb motion is weak because of low angular resolution and specular dropouts, meaning limbs that reflect the signal away from the receiver. A veiled camera keeps fine limb detail on nearby people. Rain streaks, however, inject frame-to-frame flicker into its keypoints, and it loses everything at distance. One scalar weight per sensor cannot express this, but a weight per band can.

Relation to Prior Work

Two papers show that rain veiling is a structured, modelable degradation rather than generic noise, and they supply our optical mechanism. RainFlow builds veiling-invariant and streak-invariant features for optical flow. It works on dense 2D pixel motion between frame pairs and does not handle articulated 3D bodies or radar. Dual-Rain removes rain and accumulated haze from video with two teachers that update at different rates, using depth-conditioned rain synthesis. It restores pixels and does not track people. Both rely on monocular depth for the veil. Our proposal replaces that with radar range, which removes a known failure point.

Three lines of work show that radar degrades when people are close together, and they supply our radar front-ends. RFMamba (state-space model over RF spectrograms), WiFi-Mamba, and RAPTR all handle multiple people. RFMamba reports error growing with occupancy, and WiFi-Mamba is tested only up to three people. milliMamba uses two orthogonal radars against specular dropouts. Their "frequency" means radio frequency, not motion frequency. All of them output frame-level coordinates from radar alone, and none uses a camera as a complementary witness.

Two benchmarks constrain the design by showing that radar is robust to weather but imperfect. mmBody reports depth-camera error rising from about 5.5 cm to 24.3 cm in smoke while radar stays near 8.7 cm. It also reports that occlusion raises radar error by up to 56% and that a radar frame has only about 1,000 points. RT-Pose records rainy sequences with RGB, LiDAR, and raw 4D radar. Both treat weather as an experimental condition rather than a model input. mmDiff improves radar pose with diffusion denoising and short temporal convolutions, but only for a single subject.

The motion representation comes from Cai et al.. They predict whole trajectories as DCT coefficients, which avoids error compounding frame by frame, and they decode from the torso outward to the limbs. The constraint here is the Gibbs ringing that the file notes for global transforms on abrupt motions, together with WaMo's finding that deep wavelet splits hurt. We therefore use short overlapping windows of about one second with a modest number of bands rather than one global transform.

Research Program

We would build four things. First, the testbed. We render a physical veil onto real RGB frames as $I = B t + A(1 - t)$, using measured depth where available (mmBody's Kinect) and radar range per person otherwise. Rain streaks are added on top, and $\beta$ is swept. Separately, we create multi-person radar by adding the raw complex radar tensors of single-person RT-Pose sequences recorded from the same sensor pose, placing subjects at controlled separations. This relies on the superposition principle. It deliberately ignores mutual shadowing and inter-person multipath, so it is a lower bound on difficulty and must be validated against real multi-person captures. The result is a grid with two dials, fog density and crowd spacing.

Second, the per-sensor estimators. These are an off-the-shelf 3D keypoint lifter on RGB and a radar pose network with an RFMamba-style backbone. Each emits per-person trajectories over a window, which are converted to DCT coefficients.

Third, the physical reliability inputs. We estimate $\beta$ from the contrast of each person's image region relative to the background at the radar-reported range. We compute $s_m$ from radar detections.

Fourth, the fusion and refinement. This is the weighted DCT fusion above, followed by a small torso-to-limb refinement in the style of Cai et al. Identity association across windows uses the low-frequency coefficients, which carry position and heading, together with radar Doppler.

Baselines

The naive reference is radar alone, using the same radar network that feeds our fusion and trained on the same data. mmBody and RT-Pose suggest that radar barely notices rain. Radar alone therefore isolates how much the camera adds once people crowd together. On real data, the anchor numbers from the file are an MPJPE of 9.91 cm for HRRadarPose on RT-Pose (vs. 18.43 cm for RF-Pose3D), reported by RT-Pose, and roughly 8.2 cm for P4Transformer on mmBody in standard conditions and 8.7 cm in smoke, reported by mmBody. mmDiff reports a 12.8% improvement over P4Transformer. Camera alone, with and without Dual-Rain restoration in front, is the second naive point.

The strongest competitor is an end-to-end learned camera–radar fusion transformer. It takes both feature streams, uses cross-attention, and is trained with the same rain and superposition augmentations. It has more capacity than our method and can, in principle, learn reliability from data. No reported result found on Lacuna for camera–radar fusion pose under rain on either benchmark. We will therefore train it ourselves with the same backbones, training data, number of epochs, and hyperparameter search budget as our method. On the in-distribution grid, matching it is acceptable. Beating it means lower MPJPE and fewer identity switches on held-out fog densities and spacings outside the training range.

Three ablations isolate the key ingredient. Band-flat fusion uses the same physical weights but one weight per sensor across all frequencies. Time-domain fusion is a Kalman smoother whose per-frame noise covariances come from the same $(t_m, s_m)$ model. Learned-weight fusion uses the same DCT fusion with weights predicted from image and radar features instead of physical quantities. A further ablation replaces radar range with monocular depth inside $t_m$.

Why not just…?

Why not just train an end-to-end fusion network with heavy weather augmentation? This is the real threat. Inside the training distribution it may well match us. The disagreement is about extrapolation. A network that has seen fog up to some density has no reason to lower camera trust correctly at twice that density. A weight that is a monotone function of $t = e^{-\beta d}$ should behave correctly there, because the veil physics is the same. The held-out-dial experiment below settles this directly. If the learned fusion extrapolates equally well, the physical conditioning is not worth its assumptions, and we would report that.

Why not just derain the video first and then run any fusion? Dual-Rain and similar restorers are built to make pixels look clean, not to keep keypoints geometrically faithful. The file records that veil removal can over-suppress bright structures. Restoration also adds latency, and diffusion restorers add more. We include restore-then-fuse as a baseline. If it closes the gap, the contribution shrinks to the radar-depth trick alone.

Why not just trust radar, since it already sees through rain? mmBody supports this for one person in smoke. RFMamba's error growth with occupancy, and the missing results for four or more people, suggest it breaks down at close spacing. Radar-alone error plotted against $s_m$ on the superposition grid shows exactly where the camera starts to pay off.

Why not just smooth radar trajectories in time with a Kalman filter? Smoothing removes jitter but cannot recover a limb the radar never saw. It also cannot undo an identity swap between two merged reflections. The Kalman ablation, given the same physical covariances, tests whether working in the frequency domain adds anything beyond physical weighting.

Proposed Experiments

The main experiment trains on the testbed grid with $\beta$ and crowd spacing restricted to the lower two-thirds of their ranges and tests on the upper third. We report MPJPE, per-DCT-band error, acceleration error (a jitter measure), and identity-switch rate. Results are also broken down by person range, to test the veil prediction, and by $s_m$, to test the interference prediction.

The first decisive check is the size of our margin over the learned fusion transformer on the held-out dials. A second check is whether band-dependent weights beat band-flat ones. The prediction is that the gain concentrates in the high bands for nearby people and in the low bands for crowded people. A third check is whether radar range beats monocular depth inside $t_m$ as fog thickens.

Real-data validation uses rainy RT-Pose sequences and mmBody's simulated-rain and smoke scenarios, comparing against the reported numbers above. Multi-person real checks use whatever genuinely co-present subjects RT-Pose contains, to measure how much the superposition testbed underestimates mutual shadowing. If real multi-person error is far worse than the superposition grid predicts, the radar reliability model needs a shadowing term before the claim holds.