Cameras give dense, sharp depth estimates, but a single camera cannot tell how far away things really are, so its depth "breathes" from frame to frame. Radar gives true distances and works in fog and rain, which makes it the obvious source of fixed reference points ("anchors") for pinning camera depth to the right scale. The catch is that radar lies in a structured way. When a radio wave bounces off a wall or guardrail before or after hitting a car, the sensor places a phantom copy of that car, a "ghost," deeper in the scene. That phantom moves in lockstep with the real car, so it survives the temporal filters that remove ordinary noise. Any method that trusts radar anchors will bend correct camera surfaces toward these phantoms.
The core proposal is to let the camera decide which radar returns to believe. A ghost almost always sits behind a surface the camera can see: behind a wall, a guardrail, or the road. A real return sits on a visible surface. Once camera depth is roughly aligned to metric scale, a return that lands in space the camera knows is blocked is suspect. If its mirror image across that blocking surface lands on a real tracked object with matching radial velocity, it is very likely a reflection. We combine these two tests with the scale alignment in a single estimation over a short video window. The output is temporally stable metric depth plus a per-return label: real, ghost, or ghost that has been traced back to an object hidden from the camera.
Core Claim
Recent radar–camera depth methods treat ghost removal and temporal stability as separate problems, and neither uses the geometry the camera already provides. Zhao et al.'s locally weighted scale alignment (Toward 3D scene reconstruction from locally scale-aligned monocular video depth) fits a spatially varying scale and shift to monocular depth from 25–100 sparse metric anchors. It cuts relative error by 48% on KITTI, but it assumes the anchors are clean. R4Dyn uses radar to supervise the depth of moving cars, and the file notes that a ghost hit on a vehicle actively pushes depth toward the wrong range. The ghost-aware systems either prune radar returns as generic outliers or need a known map:
- SA-RCD learns a confidence score for each radar–camera association.
- RC-PDA rejects returns whose radar motion disagrees with camera optical flow.
- mmTunnel traces ghosts back through planar tunnel walls it knows in advance.
Our claim is that the reflecting surfaces mmTunnel takes from a map can instead be estimated, imperfectly but well enough, from the camera's own aligned video depth. This turns free-space checking and mirror checking into a general, map-free test that works in open roads and cities. We expect this test to separate ghosts from real returns better than appearance- or flow-based association confidence. The reason is that ghosts are temporally coherent: they defeat flow checks, as Yataka's pseudo-tracklet analysis summarized in the file indicates. Geometrically, however, ghosts are still inconsistent with what the camera sees.
The four directions play different roles here.
- Multipath ghosting is the core mechanism.
- Temporal depth inconsistency is the target output.
- Dynamic scenes act as a constraint. The static background and each tracked moving object get separate scale fields, because a single field across a moving car repeats the smearing problem the file attributes to video depth methods.
- Geometric incompleteness enters narrowly and honestly. Ghosts traced back to their source give range evidence about objects the camera cannot see. We report this as recovered detections, not as completed shapes.
Research Program
Depth source. We start from a temporally consistent relative video depth estimator, DVD. It turns a 1.3-billion-parameter video diffusion transformer into a single-pass depth regressor and reports zero-shot AbsRel of 6.7 on KITTI. Its output carries no metric scale, so it fills the slot that our alignment makes metric. A lighter monocular backbone is a second configuration for the latency question.
Joint alignment and labeling. Over a window of roughly one second, we jointly estimate two things. The first is a local scale-and-shift field for the static background and for each tracked moving object, in the spirit of Zhao et al. The second is a weight $w_i \in [0,1]$ for each radar return $i$, read as the probability that the return is a direct, line-of-sight hit.
The weights come from three terms:
- Range residual. How far the return is from the aligned camera surface.
- Free-space penalty. We take the camera's first visible surface along the return's azimuth wedge in bird's-eye view. This is necessary because nuScenes radar has no elevation. The penalty grows with how far the return lies behind that surface, beyond a tolerance.
- Mirror support. We fit planes to vertical structure in the aligned depth: walls, guardrails, vehicle sides. We reflect the return across the nearest candidate plane. Support is high if the reflected point lands within tolerance of a radar-tracked object whose radial velocity matches after the reflection. A ground-plane mirror handles road bounces.
Alternating between fitting the scale fields with weighted anchors and updating the weights is an expectation–maximization-style loop. It starts from a heavily robust (Huber-loss) global fit to avoid the obvious circularity, in which the alignment needs clean anchors and the labeling needs the alignment.
Known confounder. Automotive radar legitimately sees under and between vehicles, so some real returns lie behind camera-visible surfaces. The free-space term alone would wrongly reject these. The mirror term is what is meant to tell them apart: a return seen under a car has no mirror partner, while a wall reflection does. We will measure how often this confounder occurs rather than assume it is rare.
Labels for ghosts. Ghosts have no annotations, so we derive them on nuScenes and View-of-Delft from a single LiDAR sweep and the 3D boxes. A radar return is labeled a ghost if it has no LiDAR or box support within a range–azimuth tolerance and lies behind a LiDAR-observed surface. It is labeled "real but camera-occluded" if it falls inside an annotated box that is hidden from the camera. We use a single sweep, not an accumulated one. Sparse Beats Dense shows that accumulating LiDAR sweeps injects motion ghosts around moving vehicles, which would contaminate exactly the labels we need.
Baselines
Naive reference points. DVD aligned to radar with one global least-squares scale and shift per window, using all radar returns. Beside it, the same depth with no radar at all, aligned by ego-odometry scale only. These show how much radar helps when trusted blindly and how much it hurts in ghost-heavy scenes.
Direct parent. Zhao et al.'s locally weighted scale alignment applied to the same DVD depth with all radar returns as anchors. It is the strongest existing method for turning sparse metric points into stable local scale. Running it on identical depth and anchors isolates the effect of anchor selection.
Robust-fitting competitor. The same local alignment with RANSAC and with Huber or Cauchy reweighting, giving the same number of iterations and the same tolerance search as our method. This is the most direct threat, discussed below.
Learned-association competitor. A radar–camera association confidence network trained on the same nuScenes training split with the same LiDAR-derived labels, modeled on SA-RCD's RCANet. The file gives no code resource for it, so we reimplement it and report that we did. Its scores replace our geometric weights inside the same alignment. We also run RC-PDA's flow-consistency filter as the motion-based alternative.
End-to-end depth completion. Sparse Beats Dense, the strongest competitor reported in the file on nuScenes radar–camera depth. It reports MAE of 1927 mm, an 11.6% gain over prior dense-supervision methods, at 21.7 FPS. RC-PDA's MER reports 1.472 m MAE. These come from different evaluation settings in the file, so we re-run both under our protocol rather than compare published figures. No reported temporal-consistency result was found on Lacuna for any radar–camera depth method on nuScenes. So for stability, the comparison is our own re-runs.
What counts as beating them. Lower MAE than locally weighted alignment with robust fitting on the ghost-heavy subset (frames where at least 10% of radar returns carry ghost labels), with no loss on the clean subset. Lower temporal alignment error than all baselines. Ghost F1 above the learned association network's.
Ablations. We remove the free-space term, remove the mirror term, and shrink the window to a single frame. These isolate whether the gain comes from camera geometry, from reflection reasoning, or merely from temporal pooling.
Why not just…?
Why not just use robust regression and treat ghosts as outliers? This is a real threat, and it may win in sparse scenes. Its weakness is structural. Ghosts from a long wall cluster spatially and move together, so within a local alignment neighborhood they can form the majority. RANSAC then fits the ghost surface. We will settle this empirically by stratifying results by the fraction of ghost-labeled returns per local neighborhood. If robust fitting matches us even where ghosts are the local majority, the free-space term is not earning its place, and the contribution reduces to mirror-based recovery.
Why not just learn a ghost classifier, as RadarDistill or SA-RCD do? RadarDistill teaches a radar-only network to suppress activations where a LiDAR teacher sees empty space. It reaches 20.5% mAP on nuScenes detection, which shows that learned suppression works for detection. A learned filter, however, sees only appearance and local context. It cannot express "this point is behind that wall in this particular scene," and it throws away returns that could reveal hidden objects. The learned-association baseline above uses identical labels and data. If it matches our ghost F1, we will report that the geometry adds interpretability and recovery but not accuracy.
Why not just rely on temporal or flow consistency? The ghost-motion evidence summarized in the file argues against it: reflections of a car moving parallel to a wall share its velocity and form coherent pseudo-tracklets. RC-PDA's flow check also depends on optical flow, which degrades at night and in rain. We report all metrics separately for the night and rain scenes of nuScenes, where we expect this baseline to fall furthest behind.
Proposed Experiments
Main benchmark. The primary benchmark is nuScenes, with its 2D Continental radar, six cameras, and LiDAR for evaluation. We report the following:
- Depth MAE and RMSE against single-sweep LiDAR, at 50 m and 80 m caps.
- Ghost precision, recall, and F1.
- A temporal alignment error for static background, computed by warping aligned depth between frames with ground-truth ego pose and measuring disagreement at LiDAR-verified static points.
Elevation. View-of-Delft's 4D radar tests whether elevation, which K-Radar shows adds 7.32% AP3D in detection, makes the ground-bounce test sharper.
Adverse weather. K-Radar's adverse-weather sequences test whether the method degrades gracefully when the camera's free-space estimate itself becomes unreliable. A decreasing ghost F1 there would be expected; a collapse would mark a limit.
Hidden-object recovery. Every ghost that is traced back to a source and lands inside an annotated, camera-occluded box counts as a recovered detection. mmTunnel reports occluded-vehicle recall rising from 22.8% to 85.7%, but in mapped tunnels. We expect far lower numbers on open roads and will report them alongside the no-recovery baseline rather than against mmTunnel.
Decisive check. The ghost-heavy-neighborhood comparison against robust local alignment decides the question. If camera free space does not beat RANSAC where ghosts outnumber real returns locally, the central claim fails.
References
- Toward 3D scene reconstruction from locally scale-aligned monocular video depth
- R4Dyn: Exploring Radar for Self-Supervised Monocular Depth Estimation of Dynamic Scenes
- Structure-Aware Radar-Camera Depth Estimation
- Radar-Camera Pixel Depth Association for Depth Completion
- Ghost Points Matter: Far-Range Vehicle Detection with a Single mmWave Radar in Tunnel
- DVD: Deterministic Video Depth Estimation with Generative Priors
- Sparse Beats Dense: Rethinking Supervision in Radar-Camera Depth Completion
- RadarDistill: Boosting Radar-based Object Detection Performance via Knowledge Distillation from LiDAR Features