A camera moving through a room can already produce smooth depth video. Depth predictors built on large image models, such as Video Depth Anything and its streaming version Online Video Depth Anything, no longer flicker. Yet stacking their output into a 3D model still goes wrong. The standard tool for building surfaces from depth is the truncated signed distance function (TSDF), a voxel grid in which each cell stores its distance to the nearest surface. A TSDF averages away random noise but not systematic error. If a predictor's scale slowly drifts, or its error bends differently in different parts of the image, the same wall gets fused as several parallel copies. The 3D map the system is building is the most stable record of the scene so far, but no current video depth model ever looks at it. Every pipeline is one-way: depth goes into the map and nothing comes back.
We propose to close that loop. Before each new depth frame is fused, we render depth and fusion confidence from the current TSDF into the new camera view. We then use the confidently mapped pixels to correct the frame with a per-pixel scale and shift, and only then fuse it. The question is not whether feedback can help, which is plausible. The question is when it helps, how it avoids locking in its own early mistakes, and whether a learned correction does any better than a closed-form fit against the map.
Core Claim
Leftover video depth error comes in two parts: slow drift in scale and shift over time, and error that varies across the image. Both can be removed online, without any external metric anchor points. The anchors come from the persistent TSDF itself, but only where the TSDF is trustworthy. A cell is trusted when it has high accumulated weight, has been seen from several viewpoints, and has not been carved away, meaning later frames looked straight through it.
This differs from four nearby lines of work. Toward 3D scene reconstruction from locally scale-aligned monocular video depth showed that per-frame global alignment leaves surface-warping errors. It fixed them with locally weighted linear regression against 25–100 external metric points from SLAM, structure-from-motion or range sensors, and then fused once with a standard TSDF. We keep its insight that error varies across the image but replace the external anchors with the map, which makes the method a streaming loop rather than a single pass. NeuralRecon keeps recurrent 3D memory, but it predicts the TSDF directly and argues that "predict depth, then fuse" is inherently flawed. FineRecon reaches a far higher ScanNet F1 (75.5 against NeuralRecon's 0.562) by leaning on 2D depth, but its depth comes from a multi-view stereo network with no temporal handling and no feedback from the volume. AMB3R pushes features into a sparse voxel backend so that each 3D point is represented once across views. It processes clustered image batches and predicts one scale per scene, not per-pixel corrections in a stream.
Research Program
The system has four parts, and all of them are kept fixed across comparisons. The depth source is a frozen Online Video Depth Anything, chosen because it is causal and runs at 42 FPS on an A100. The map is a voxel-hashed TSDF, which stores memory only for blocks near surfaces, as in TANDEM. Poses come either from ground truth, which isolates the depth question, or from a SLAM tracker of the kind NeuralRecon uses. Fusion weights follow the inverse-variance idea of Probabilistic Volumetric Fusion for Dense Monocular SLAM, so that confidence can be compared across frames.
At frame $t$, we ray-cast the map into the new camera view to get a rendered depth $\hat D_t$ and a confidence $c_t$, which combines accumulated weight and the number of distinct viewpoints. The raw predicted depth is $d_t$. We then estimate correction maps $s_t(p)$ and $b_t(p)$ for each pixel $p$, so that
$$D_t(p) = s_t(p)\, d_t(p) + b_t(p),$$
and fuse $D_t$.
We study two estimators. The analytic estimator fits $s_t, b_t$ by locally weighted linear regression against $\hat D_t$, weighting each pixel by $c_t$ and by spatial proximity. It is the method of Toward 3D scene reconstruction..., with map-derived anchors in place of sensor anchors. The learned estimator is a small convolutional head. It takes as input the predictor's frozen decoder features, $d_t$, $\hat D_t$, $c_t$, and a mask of pixels the map has never seen, and it outputs $s_t, b_t$. We train it on unrolled ScanNet training sequences, with the map built online during training, gradients stopped through the map, and a loss on metric depth error. The learned head can do things a regression cannot. It can stop trusting the map at occlusion boundaries and on objects that move. It can extend corrections into regions the map has not yet seen using image content. It can respect depth edges that a smooth regression would blur, which is the boundary failure documented by Revisiting Depth Representations for Feed-Forward 3D Gaussian Splatting.
The central risk is self-confirmation. Once early frames have written a distorted wall into the map, later frames can be bent to agree with it. Three mechanisms counter this. Anchors are used only where several viewpoints agree. Carving can lower a cell's confidence whenever new frames see through it. The confidence threshold is a measured parameter, not a hidden one. To initialise metric scale, we report two settings: a single scale per sequence, fixed once from the first keyframe, which matches standard affine-invariant evaluation, and a fully metric setting that uses the global scale predicted by MapAnything.
Baselines
The naive reference point is per-frame image depth plus TSDF. Each frame's depth from the frozen image encoder alone is aligned with one per-sequence scale and fused. This shows how much of the problem temporal models have already solved.
The strongest "do nothing extra" competitor is Online Video Depth Anything fused directly into the same TSDF with the same poses, voxel size and weighting. It is the most likely rival because it is already causal, fast and temporally stable. Beating it means a higher ScanNet mesh F-score and a lower rate of doubled surfaces at equal frame rate. No reported result found on Lacuna for mesh F-score of oVDA or VDA depth fused into a TSDF, so we will measure it ourselves.
The analytic closed loop described above serves as both a baseline and an ablation. It shares everything with the learned head except the estimator, so it isolates whether learning matters. A cruder loop, with one global scale and shift per frame fitted against the map, tests whether per-pixel correction matters at all.
The anchor-based upper reference is Toward 3D scene reconstruction... given 100 ground-truth sparse anchors per frame. That paper reports Relative Error dropping 54% on ScanNet and 48% on KITTI. Getting close to it without external anchors would be the headline result.
For volumetric references we cite FineRecon (F1 75.5, Chamfer 5.18 on ScanNet) and NeuralRecon (F-score 0.562 at about 33 FPS). FineRecon uses multi-view stereo depth and heavy dense volumes, so we compare to it as a quality ceiling and to NeuralRecon as the matched online competitor. Where we rerun methods, they use the same evaluation meshes and voxel resolution.
A learned head that receives no map input but is otherwise identical isolates the feedback from simply adding trainable capacity. All learned variants get the same training data and number of tuning trials.
Why not just…?
Why not just fuse oVDA depth into a standard TSDF? Temporal smoothness is not the same as 3D consistency. Online Video Depth Anything itself acknowledges scale drift on very long sequences. Toward 3D scene reconstruction... shows that error varying across the image survives any global correction and warps fused surfaces. This is exactly our strongest baseline. If it matches the loop on long ScanNet and TUM sequences, the proposal fails, and we will report that.
Why not just run LWLR with SLAM sparse points as anchors? This is a real option whenever a SLAM backend already exists, and we include it. Its weaknesses are density and reliability. Sparse SLAM points cluster on texture and vanish on the white walls and ceilings where Probabilistic Volumetric Fusion leaves holes. The map, by contrast, offers dense anchors wherever it has already converged. We will run SLAM-anchor LWLR and map-anchor LWLR side by side, and also their combination, so that if sparse anchors suffice this becomes clear.
Why not just render the map and fit a closed-form correction — why learn anything? This is the most serious threat, and we treat it as the default answer. The learned head is justified only if it beats the analytic loop on boundaries, on dynamic content (Bonn), or in newly seen regions. We report results split by those pixel categories. If the analytic loop wins overall, the contribution reduces to the confidence-gated closed loop itself, which is still new relative to every one-way pipeline in this literature.
Why not just predict the TSDF directly, as NeuralRecon does? On the evidence collected here, direct volumetric prediction trails depth-guided methods by a wide margin on ScanNet (0.562 against 75.5 F1). Our wager is that the depth-first approach failed earlier because of inconsistent depth, not because depth maps are the wrong intermediate.
Proposed Experiments
The main evaluation is ScanNet mesh reconstruction, reporting F-score, Chamfer distance, precision and recall at matched voxel size, in two settings: ground-truth poses and SLAM poses. We add one diagnostic tailored to the failure we target, the doubled-surface rate. For each ground-truth surface point we march along its normal and record the fraction of points whose fused TSDF crosses zero more than once within the truncation band. This measures ghosting directly rather than through averages.
We also track depth error of the corrected frames (AbsRel, $\delta_1$) on ScanNet, KITTI and Bonn, along with the temporal consistency measures used by StableDPT. All of these are reported against sequence length, to show whether drift is bounded or merely slowed.
The decisive checks follow from the claims. Feedback must beat the matched learned head without map input on long sequences; otherwise feedback is not the active ingredient. Per-pixel correction must beat the per-frame global loop on the doubled-surface rate. The method must not self-confirm. To test this, we inject a deliberate scale error into the first 50 frames and measure whether the map recovers or freezes it as the threshold on $c_t$ varies. On Bonn's moving people, the confidence and carving gate must prevent the map from imprinting transient objects, measured by error on pixels covered by dynamic masks. Finally, the whole loop must stay within a small fraction of oVDA's frame rate, or the claim of real-time online fusion does not hold.
References
- Video Depth Anything: Consistent Depth Estimation for Super-Long Videos
- Online Video Depth Anything: Temporally-Consistent Depth Prediction with Low Memory Consumption
- Toward 3D scene reconstruction from locally scale-aligned monocular video depth
- NeuralRecon: Real-Time Coherent 3D Reconstruction From Monocular Video
- FineRecon: Depth-aware Feed-forward Network for Detailed 3D Reconstruction
- AMB3R: Accurate Feed-forward Metric-scale 3D Reconstruction with Backend
- TANDEM: Tracking and Dense Mapping in Real-time using Deep Multi-view Stereo
- Probabilistic Volumetric Fusion for Dense Monocular SLAM
- Revisiting Depth Representations for Feed-Forward 3D Gaussian Splatting
- MapAnything: Universal Feed-Forward Metric 3D Reconstruction
- StableDPT: Temporal Stable Monocular Video Depth Estimation