Doppler-Supervised Motion for Dynamic Gaussian Reconstruction from a Single Camera

A single moving camera sees sideways motion well and motion toward or away from itself badly. A car approaching head-on barely shifts in the image; it only grows slowly. Automotive radar has the opposite blind spot. Its angular resolution is coarse, but every return carries a Doppler shift: a direct, instantaneous measurement of how fast that surface moves along the line between sensor and target. When radar and camera are mounted close together, the radar's line of sight nearly coincides with the camera ray. The velocity component the camera cannot observe is then exactly the one the radar measures. This proposal fits a dynamic 3D scene model to camera video and uses radar Doppler as a direct supervisory signal on the model's motion field. The camera constrains motion across the image, radar constrains motion along the ray, and together they pin down full 3D velocity.

The scene model is a set of 3D Gaussians: small colored ellipsoids that can be rendered quickly into images, a technique called 3D Gaussian splatting. A small network moves each ellipsoid over time. This kind of single-camera dynamic reconstruction is known to drift and collapse along the depth axis. A side benefit of the radar supervision is that static and moving surfaces separate on their own. A static point's Doppler is fully predicted by the vehicle's own motion, so any leftover Doppler marks something that moves independently. That gives a decomposition of the scene into background and movers without a video tracker or bounding boxes.

Core Claim

The central claim is that radial-velocity supervision from Doppler, not radar range points, is what fixes depth-axis motion drift in monocular dynamic reconstruction. A model supervised this way should produce object trajectories with much lower error along the camera's optical axis, the depth direction. It should also render far more stably from viewpoints away from the driving path than camera-only dynamic Gaussians or camera models given radar positions as depth hints.

Let Gaussian $i$ have center $\mathbf{p}_i(t)$ and velocity $\mathbf{v}_i(t)=\partial_t \mathbf{p}_i(t)$, both produced by the deformation network. Let the radar sit at $\mathbf{o}(t)$ with ego-velocity $\mathbf{u}(t)$. The radial velocity the radar would measure from that Gaussian is

$$\hat r_i(t) = (\mathbf{v}_i(t)-\mathbf{u}(t))\cdot \frac{\mathbf{p}_i(t)-\mathbf{o}(t)}{\lVert \mathbf{p}_i(t)-\mathbf{o}(t)\rVert}.$$

For each radar return $k$ with measured Doppler $r_k$, the Doppler loss compares $r_k$ to a weighted average of $\hat r_i$. The weights $w_{ki}$ come from a soft kernel in range, azimuth and elevation that matches the radar's resolution, and they are scaled by each Gaussian's opacity. The total training objective is

$$\mathcal{L} = \mathcal{L}_{\text{photo}} + \lambda_D \sum_k \rho\Big(r_k - \sum_i w_{ki}\hat r_i\Big) + \lambda_R \mathcal{L}_{\text{range}},$$

where $\rho$ is a robust penalty that limits the influence of multipath ghost returns, $\mathcal{L}_{\text{range}}$ is a weak radar-range term, and the $\lambda$ weights balance the three terms. The photometric term constrains tangential motion through image appearance, and the Doppler term constrains the radial component. Static and dynamic labels are not supervised. They come from fitting the ego-motion-only Doppler prediction ($\mathbf{v}_i=0$) and testing which Gaussians the static explanation fits poorly.

Relation to Prior Work

Two lines of work motivate the gap.

Monocular dynamic reconstruction already knows its own failure mode. Deformable 3D Gaussians introduced the setup we reuse: canonical Gaussians plus a time-conditioned network that moves them. Its authors report that it degrades under restricted camera trajectories, which is exactly the forward-driving case. USplat4D diagnoses the problem precisely, showing that uncertainty along the line of sight dominates. It responds with an uncertainty-weighted graph in which uncertain Gaussians follow reliable "key" Gaussians. Its authors concede that the method has no physical anchor when the visual motion prior fails. We replace the heuristic anchor with a measured one.

Radar work shows that Doppler carries geometry but has stayed static or per-frame. DART uses Doppler to collapse radar's elevation ambiguity into thin rings for static tomography. It explicitly assumes that all Doppler comes from sensor motion, so moving objects break it. Our model resolves that tension by giving every Gaussian its own velocity, so object motion is explained rather than mistaken for geometry. RadarSplat supplies useful machinery: radar-aware Gaussian opacity and handling of multipath noise. It is static-only and radar-only. CRAB uses radar occupancy, Doppler and reflection strength to clip "depth trails" along camera rays. RaGS fuses 4D radar with monocular depth inside Gaussians. Both are single-frame 3D detectors; RaGS's authors state that it processes frames independently. Neither one uses Doppler as a constraint on a continuous motion field.

DIO builds decomposable 4D occupancy and flow, but from LiDAR alone, with no camera input and with box guidance during training.

Some prior work constrains the design. RobustSplat shows that the routine that adds new Gaussians where error is high, called densification, turns transient motion into floating mid-air Gaussians. It also shows that switching densification off over-smooths the scene. We therefore gate densification by Doppler residual rather than delaying or disabling it: Gaussians whose motion is radar-confirmed may grow early. Split4D shows that tracker-free decomposition beats tracker-based masks (0.893 versus 0.592 mean intersection-over-union on Multi-Human). It also documents that vision-only decomposition fails under occlusion. That is the setting where radar's independent velocity signal should help.

Research Program

We would build on a Deformable-3DGS-style model with an explicit velocity output and add four pieces.

The first is a differentiable Doppler renderer. It takes the model's per-Gaussian radial velocities and produces predicted Doppler values at each radar return location. The soft association kernel has a learnable width, initialized from the sensor's angular resolution.

The second is ego-motion handling. The radar's own velocity $\mathbf{u}(t)$ comes from dataset poses, and we also test it estimated from the static-return Doppler consensus, so that sensor timing or mounting errors do not leak into object motion.

The third is Doppler-gated densification. A region may densify early only if its photometric error coincides with a nonzero Doppler residual. That separates real movers from shadows and exposure changes, which have no radar signature.

The fourth is an analysis of when the method can work. Doppler only constrains motion along the radar ray. The camera–radar baseline and the scene geometry therefore set how much depth-axis information Doppler adds, and we will measure the gain as a function of the angle between the camera ray and the radar ray.

Data comes from camera plus 4D-radar driving datasets: View-of-Delft as used by RaGS, TJ4DRadSet, and the K-Radar adverse-weather sequences. We use nuScenes radar as a sparser stress test. LiDAR is withheld from training on all of them and used only as ground truth.

Baselines

The naive reference is static 3D Gaussian splatting trained on the same clip. It shows how much damage moving actors do, in floaters and smeared vehicles. We expect any dynamic method to beat it easily, so it calibrates the scale of the problem rather than serving as a real competitor.

The main camera-only competitor is Deformable 3D Gaussians. It is the direct parent architecture, so comparing against it isolates the effect of adding radar. It uses identical initialization, iteration count and resolution; the only difference is that our Doppler and range terms are added.

The strongest existing method on the depth-ambiguity question is USplat4D. It targets exactly the line-of-sight drift we target, without extra sensors. Its reported number is 19.63 PSNR (peak signal-to-noise ratio) on DyCheck, against 19.32 for MoSca. DyCheck has no radar, so this number cannot be compared directly. We found no reported result on Lacuna for any dynamic Gaussian reconstruction method on View-of-Delft, TJ4DRadSet or K-Radar, so we will run USplat4D ourselves on our clips. It gets the same frames, the same monocular depth and optical-flow priors it normally consumes, and a hyperparameter sweep matching ours in size.

Two radar-aware ablations isolate the key ingredient. The first, "radar as points," feeds radar returns only as sparse depth supervision through the range term, with no Doppler. This is the natural reading of how RaGS and CRAB use radar geometry. The second, "radar points plus monocular depth foundation prior," aligns a pretrained depth predictor's output to radar returns per semantic region, in the style of MS-GS.

A LiDAR-supervised dynamic Gaussian model is an upper reference, not a target. Beating the strongest competitors means two things on held-out frames: a clear reduction in trajectory error along the optical axis against USplat4D and against the radar-points ablation, and higher novel-view PSNR at laterally shifted viewpoints, with no loss in on-path rendering quality.

Why not just…?

Why not just use a monocular depth foundation model and run dynamic Gaussians? Such models predict per-frame depth whose scale is not consistent across frames. MS-GS needed per-region alignment to sparse metric anchors just for static scenes. More fundamentally, a depth map says where a point is, not how fast it moves along the ray. Differencing noisy depths across frames amplifies exactly the error we care about. The foundation-prior baseline above tests this directly. If it matches Doppler supervision on depth-axis trajectory error, our claim fails.

Why not just use radar points as depth anchors and skip Doppler? This is the most serious threat. Radar range returns are metric, and they might fix scale well enough that velocity follows from smooth trajectories. Our counter-argument is that radar positions are noisy in angle by several degrees, so anchoring alone leaves objects jittering. Doppler, in contrast, measures the derivative directly and accurately. We do not want to argue this; the radar-points ablation settles it. If Doppler adds nothing beyond positions, the proposal reduces to multimodal depth supervision, and we would report that honestly.

Why not just use LiDAR, which gives much better geometry? On datasets that carry LiDAR, LiDAR-supervised reconstruction will likely win on clear-weather geometry, which is why it is our upper reference. But LiDAR gives no instantaneous velocity. K-Radar reports LiDAR detection falling to 29.7% 3D average precision (AP3D) in heavy snow while radar keeps 44.5%. On K-Radar's snow and night splits, we will check whether the radar-Doppler model narrows or reverses the gap to LiDAR on depth-axis error.

Why not just use the uncertainty graph of USplat4D? Its key Gaussians are chosen from photometric convergence. Where image motion is uninformative, as with head-on approaching cars or textureless trailers, the anchors themselves are wrong, as its authors acknowledge. We will split evaluation by object heading relative to the camera. Our prediction is that the gap over USplat4D is largest for objects moving along the ray and near zero for crossing traffic. If both methods perform the same on the along-ray slice, that would falsify the mechanism.

Proposed Experiments and Decisive Checks

All methods are trained per clip on camera plus radar from View-of-Delft, TJ4DRadSet and K-Radar. Every few frames are held out, and LiDAR is reserved as ground truth.

Geometry is scored as depth error against held-out LiDAR, split into near and far ranges. Motion is scored against velocities of annotated objects derived from their 3D boxes across frames, reported separately for the along-ray and cross-ray components. Rendering is scored with PSNR, SSIM (structural similarity) and LPIPS (a learned perceptual distance) on held-out timestamps. We also render at virtual viewpoints shifted sideways from the trajectory, scored against LiDAR-projected depth since no real images exist there. Decomposition is scored as intersection-over-union between Gaussians our model labels dynamic and LiDAR points inside annotated boxes, compared against Split4D-style contrastive clustering on the same clips.

The decisive check is the heading-stratified ablation. The prediction is that Doppler supervision cuts along-ray velocity and depth error substantially relative to both the radar-points ablation and USplat4D, while leaving cross-ray error roughly unchanged. Two further results would support the mechanism. The first is a dose-response curve: degrading the Doppler signal with added noise or reduced radar density should degrade along-ray accuracy gracefully, not collapse it. The second is that Doppler-gated densification should remove shadow-induced floaters that RobustSplat-style delayed densification leaves in place, measured as floater count in free space confirmed empty by LiDAR. If the along-ray gain disappears once radar positions are supplied, we will conclude that Doppler adds nothing beyond positions.