A camera alone cannot tell how far away things are. A single image fits infinitely many 3D scenes, so depth networks guess a different overall scale for every frame. Run frame by frame on video, the result flickers and "breathes." A cheap, sparse laser scanner (LiDAR, which measures distance by timing its own light pulses) fixes the scale and works in darkness, but it covers only a few percent of pixels. In rain, fog, or snow it also returns spurious points from airborne droplets. This proposal rests on one observation about how these two failures differ over time. Real surfaces return laser hits again and again as the vehicle moves, while weather clutter and sensor noise appear at random places and vanish. If we keep a causal, world-fixed memory of laser points weighted by how persistently each one is re-observed, that single statistic does three jobs at once. It strips weather clutter, separates static structure from moving objects, and gives every video frame the same metric anchors. Frames that agree with the same world-fixed anchors agree with each other. Temporal consistency then comes from geometry rather than from image matching, and image matching is exactly what breaks at night and in fog.
Concretely, we build a small causal module. It sits on top of a frozen monocular depth foundation model and reads a persistence-weighted map accumulated from few-beam LiDAR sweeps. It outputs metric depth that should stay stable over time across day, night, fog, rain, and snow. The claim we test is narrow. Existing methods make depth metric or make it temporally stable, and they do so using only the current sweep or using camera appearance. A persistence-weighted memory should beat both on stability and accuracy under adverse conditions, using only past frames.
What Exists and What Is Missing
Sparse-LiDAR monocular depth is well established, but it works one frame at a time. LiDARTouch feeds 4-beam LiDAR into a depth completion network and replaces the learned pose network with a feature-matching pose solver anchored to laser points. It reaches 2.504 m RMSE on KITTI and eliminates the "infinite-depth" failure, in which a car moving with the ego vehicle is predicted to be at infinity. FusionDepth dilates the sparse points into a pseudo-dense input and refines depth in 3D. Both discard each sweep after use. Both rely on image photometry or image features, which AURORA-KITTI shows fail together with LiDAR in bad weather. These papers supply our input setting (4-beam LiDAR plus RGB), and they motivate the gap: un-hit pixels stay purely monocular and nothing ties consecutive outputs together.
Weather robustness is likewise per-frame. AURORA-KITTI's DDCD distills a clean-image Depth Anything V2 teacher into a student that sees corrupted RGB and corrupted LiDAR. It reports a 35.1% RMSE reduction over zero-shot Marigold-DC. DDCD is our strongest single-frame competitor, and its benchmark is our main testbed. Its scale comes entirely from the current sweep, so heavy beam dropout leaves it unanchored. That is the specific weakness persistence is meant to address.
Temporal aggregation is mature offline but not connected to online video depth. DOC-Depth accumulates LiDAR odometry into dense static clouds and strips dynamic points by range-image visibility voting, raising KITTI density from 16.1% to 71.2%. It is an annotation tool, though, and it fails on temporarily stopped vehicles. Zheng et al. supply the principle we reuse: uncorrelated noise averages out across sweeps while real surfaces get coherent repeat hits. MSC is the closest prior work that combines all four directions. It warps the sweeps at $t-1$ and $t+1$ into frame $t$ with a visual pose module and adds depth- and photometric-consistency losses. It is non-causal, limited to three frames, depends on image features for pose and masking, and explicitly does no LiDAR denoising. We differ on each of those points.
Video depth stabilizers address flicker but not scale or weather. STATIC avoids optical flow by using surface-normal variance to lock static regions over a two-frame window. StableDPT cross-attends each frame to keyframes sampled across the whole video, including the future. Both output depth only up to an unknown scale and shift, and both are passive RGB methods. StableDPT also constrains our design. It shows a frozen foundation encoder plus light decoder layers suffices for stability, so we adopt that recipe and replace its non-causal keyframes with a causal geometric memory.
Core Claim
Weight each accumulated laser point by how persistently it is re-observed, after compensating for vehicle motion. The resulting world-fixed anchor set is simultaneously cleaner than any single sweep in adverse weather and identical across frames by construction. A depth head conditioned on this set therefore produces metric depth that is both more accurate and more temporally stable than single-sweep fusion, without any reliance on optical flow or photometric matching.
Research Program
The memory is a voxelized point map in world coordinates. It is updated causally using poses from LiDAR-inertial odometry, which Hou et al. and Helmberger et al. show stays reliable where visual tracking fails. Each voxel keeps two counts. The first, $h$, is the number of sweeps in which a return landed in it. The second, $f$, is the number of sweeps in which a laser ray passed through it to a farther surface, which is evidence that the voxel is empty. Its anchor confidence is
$$c = \left(1 - e^{-h/\tau}\right)\, e^{-f/\kappa},$$
where $\tau$ sets how many repeat hits are needed before a point is trusted and $\kappa$ sets how quickly evidence of emptiness erases it. Backscatter near the sensor rarely accumulates $h$. A stopped car that later drives off accumulates $f$ as rays pass through its old position, which patches the case DOC-Depth leaves open. Points from the current sweep enter with their own flag, so moving objects, including the "follow-me" car behind the infinite-depth failure, are still anchored instantaneously.
At each frame, the memory is rendered into the camera as a sparse depth image plus a confidence image. The previous output is warped into the current view using the odometry pose, a geometric warp in the spirit of Düzçeker et al.. The depth head receives frozen Depth Anything V2 features, the rendered anchors, the confidence, and the warped previous depth. It is a small DPT-style decoder like StableDPT's, but it uses only past information.
Training combines four terms. A metric loss against KITTI ground truth is weighted by valid pixels. A distillation term against the clean-image teacher follows DDCD. A geometric consistency term compares the current output with the pose-warped previous output on pixels whose anchors have high $c$. No term uses optical flow. Weather is applied with independent random draws per sweep and per frame, so the network cannot learn to trust clutter that happens to repeat.
Baselines
The naive reference is the frozen foundation model run per frame. Its relative depth is converted to metric units by a robust (Huber) least-squares scale-and-shift fit to the current sweep's points. Every method sees the same backbone, the same sparse input, and the same corrupted frames. This tells us how much of the problem is merely a global scale error.
The strongest causal smoothing competitor applies the same fit followed by exponential smoothing of the scale and shift over time, with the smoothing rate tuned on validation. If anything matches us cheaply, it is this. We give it the same hyperparameter-search budget as our head.
The strongest weather competitor is DDCD from AURORA-KITTI. It will be retrained on our 4-beam subsampling with the same teacher and data. The file reports its gain only as a 35.1% RMSE reduction against Marigold-DC. No reported absolute AURORA-KITTI RMSE was found on Lacuna for DDCD, so we rerun it and compare at equal input. Beating it means lower metric RMSE and lower temporal error averaged over the night and precipitation conditions, at comparable latency (DDCD reports 48.6 ms per frame).
On clear KITTI with 4 beams, the reference is LiDARTouch at a reported 2.504 m RMSE with 0.0% catastrophic distance rate. We must not regress either number. MSC is run as the existing temporal completion method; it reports 1212.69 mm validation RMSE on KITTI2012 with full LiDAR, so we compare at equal beam count by retraining. StableDPT plus a per-sequence scale fit serves as a non-causal upper reference for stability.
Three ablations isolate the key ingredient. In the first, every accumulated point gets equal weight, which tests whether persistence weighting rather than mere densification does the work. The second keeps only single-sweep anchors. The third substitutes visual odometry for LiDAR-inertial poses, and ground-truth poses give the ceiling. A further ablation removes the warped previous depth to separate the effect of the memory from that of recurrence.
Why not just…?
Why not just fit scale and shift to the current sweep and smooth them over time? This is the real threat. If the dominant error is one global scale per frame and weather is mild, it may match us. It cannot correct errors that vary across the image, and it has nothing to fit when fog or snow removes most long-range returns. Its fit is also biased by near-sensor backscatter, which a robust loss only partly rejects. The decisive comparison splits results by condition and by distance band. If smoothed affine fitting is within noise of our method on night and precipitation frames, persistence adds nothing beyond what a scale estimate provides, and we will report that.
Why not just feed the accumulated point cloud into an existing completion network such as DDCD? This is the preprocessing alternative, and we run it as a baseline with the same memory but without confidence weighting or the warped-previous input. DOC-Depth shows densification helps offline. Unweighted accumulation in weather, however, also accumulates clutter that happens to land in the same voxels, and smear from imperfect motion compensation. The equal-weight ablation measures exactly this. If it closes most of the gap, the contribution shrinks to a memory format, and we will say so.
Why not just run a video depth model like StableDPT and fix scale afterwards? StableDPT needs future keyframes, so it cannot drive a vehicle. STATIC uses two frames, and its normals degrade in darkness. Both rely on image appearance for consistency. We include StableDPT plus scale fitting as a non-causal reference. If our causal method matches its temporal metrics at night while beating it on metric RMSE, the argument holds. If StableDPT stays far more stable even in clear daytime, then geometric anchoring alone does not produce visual smoothness, and a causal attention layer would need to be added.
Proposed Experiments
The primary testbed is AURORA-KITTI's day/night × clear/fog/rain/snow conditions at three severity levels, with LiDAR subsampled to 4 and 16 beams. Clear KITTI Eigen is used for comparison with LiDARTouch. Zero-shot transfer is tested on DENSE, for which no reported numerical result was found on Lacuna. One practical check comes first. Our method needs ordered sequences with weather sampled independently over time. If AURORA-KITTI's released pairs are not sequence-ordered, we regenerate corruptions on KITTI raw sequences with its simulation protocol. We also verify which DENSE segments are sequential before claiming real-world temporal results.
Accuracy is measured with RMSE, AbsRel, δ<1.25, and catastrophic distance rate. Temporal stability uses TGM and TCC as in StableDPT, with optical-flow warping error computed on the clean paired images so weather does not corrupt the metric. We add a geometric jitter measure. It back-projects predicted depth on static ground-truth points into world coordinates using ground-truth poses and reports the frame-to-frame spread of each point's position.
The decisive checks are three. First, at night and in heavy fog, our method must beat both DDCD and smoothed affine fitting on RMSE and on world-frame jitter at the same time. Second, the equal-weight ablation must be clearly worse in precipitation than full persistence weighting, while being close to it in clear weather. That is the signature that persistence, not density, is doing the denoising. Third, catastrophic distance rate must stay at LiDARTouch's 0.0% on clear KITTI, showing that the static memory does not suppress anchors on moving objects.
References
- LiDARTouch: Monocular metric depth estimation with a few-beam LiDAR
- Advancing Self-supervised Monocular Depth Learning with Sparse LiDAR
- AURORA-KITTI: Any-Weather Depth Completion and Denoising in the Wild
- DOC-Depth: A novel approach for dense depth ground truth generation
- Seg2Box: 3D Object Detection by Point-Wise Semantics Supervision
- Self-Supervised Depth Completion Based on Multi-Modal Spatio-Temporal Consistency
- STATIC : Surface Temporal Affine for TIme Consistency in Video Monocular Depth Estimation
- StableDPT: Temporal Stable Monocular Video Depth Estimation
- SUPS: A Simulated Underground Parking Scenario Dataset for Autonomous Driving
- The Hilti SLAM Challenge Dataset
- DeepVideoMVS: Multi-View Stereo on Video with Recurrent Spatio-Temporal Fusion