Streaming Gaussian Maps That Refine Phone LiDAR Across Frames

Phones now ship with depth sensors, but what they measure is coarse. An iPhone LiDAR depth map is a few hundred pixels wide. It smears depth across object edges and fades out beyond a few meters. Two families of methods attack 3D capture from such devices, and neither fits a live scan. Methods that fit a 3D model to each scene by gradient descent, such as AGS-Mesh, clean up the sensor well but take minutes per room. Feed-forward methods, which predict a 3D scene in one network pass after training on many scenes, run in under a second. They ignore the depth sensor entirely and rely on depth guessed from RGB, which drifts in scale and flickers from frame to frame.

We propose a feed-forward model that keeps a growing 3D map made of Gaussian splats (small semi-transparent ellipsoids that can be rendered quickly). For each new frame, it predicts corrections to the splats already in view, using the new frame's coarse LiDAR, its RGB image, and a depth-and-confidence image rendered from the current map. It does not spawn a fresh layer of splats for every pixel. The central bet is that the two problems that matter after metric scale is fixed are sensor errors at edges and at long range, and frame-to-frame jitter. Both can be solved together by fusing many viewpoints inside the map. A single frame cannot do this. The network becomes a learned, recursive version of classical multi-frame depth super-resolution, and it outputs a renderable 3D scene rather than a depth image.

Core Claim

Once a metric depth sensor is present, temporal depth inconsistency is no longer mainly the global scale-and-shift drift that video depth work targets, as in DVD. Least-squares alignment to the LiDAR removes most of that cheaply. What remains is error localized at depth edges, at long range, and on thin structures. These errors are biased rather than random: LiDAR edge bleeding consistently pushes foreground depth backward. Averaging over time, as classical volumetric fusion does, therefore cannot remove them.

Different viewpoints do see these errors differently. As the camera moves, an edge that was smeared in one frame is seen cleanly from the side in another. The claim is that a network allowed to compare the incoming measurement against the accumulated map can learn which observation to trust per location. It should produce geometry that is both sharper at boundaries and more stable over time than any per-frame method, while running at streaming rates.

This differs from existing feed-forward splatting systems in what each frame contributes. DepthSplat, MonoSplat, Flash-Mono and LongSplat all take RGB only, and all treat each frame as a source of new per-pixel splats. LongSplat then prunes redundant ones afterward. Here a frame is a measurement that updates splats that already exist.

Research Program

Per-frame inputs. The model receives the RGB frame, the raw low-resolution LiDAR depth with its native confidence, and a rendering of the current map from the frame's camera. That rendering contains depth, opacity, and a per-splat uncertainty channel.

Encoder. A lightweight image encoder with frozen monocular depth features supplies relative geometry and edges, following the dual-branch design that DepthSplat showed helps in textureless regions.

Update head. For every visible existing splat, the head predicts four quantities, written out in the paragraph that follows: a displacement along the viewing ray, an opacity change, a new uncertainty, and whether the splat should be retired.

Spawn head. For pixels the map does not yet explain, a second head creates new splats. These are 2D surfels (flat disks), which Flash-Mono used to suppress floaters.

The update has a natural form. Write $z_i$ for a splat's depth along the ray and $\sigma_i$ for its uncertainty. Write $d$ for the fused depth the network proposes for the pixel the splat projects to, and $\sigma_d$ for its predicted uncertainty. The model is trained so that its update behaves like a learned, uncertainty-weighted correction:

$$z_i \leftarrow z_i + k_i\,(d - z_i), \qquad k_i = g_\theta\!\left(\sigma_i, \sigma_d, \text{edge and range context}\right),$$

Here $k_i \in [0,1]$ is a gain the network outputs per splat. We do not hard-code a Kalman filter. Error at edges is biased, so the gain must depend on context such as distance to a depth discontinuity and viewing angle. A gain computed from variances alone would miss that dependence. The equation is mainly a lens for analysis: we will measure whether learned gains fall at edges seen head-on and rise when the same edge is seen obliquely.

Training data. We train on ScanNet++. Its iPhone RGB-D sequences come with laser-scanned ground truth at sub-millimeter accuracy, so we can render dense true depth for every phone frame. We use the provided camera poses; joint pose estimation is out of scope for the first study.

Losses. Training combines four terms:

  • a rendering loss on held-out frames;
  • a depth loss against the laser-derived depth;
  • a temporal term penalizing changes in rendered depth of static surface points between consecutive updates, beyond what the ground truth changes;
  • a one-directional Chamfer loss from splat centers to the laser surface, the 3D-space supervision that the PM-Loss work found fixes boundary tearing.

We use the laser surface in place of that work's VGGT pointmaps, so no teacher errors leak into training. Training uses truncated sequences of 30 to 60 frames, with back-propagation through the update chain, so the network learns to correct its own earlier mistakes.

AGS-Mesh supplies a key method component. Its depth-normal consistency check compares normals backprojected from LiDAR against monocular normals to reject edge-bleeding samples. We compute that disagreement as an extra input channel. The network can then learn the filter, with no optimization loop needed.

Baselines

Raw sensor floor. The naive reference point is direct unprojection of each raw LiDAR frame into splats, plus classical volumetric fusion (a truncated signed distance field, which averages depth over time) of the same LiDAR. This shows how much any learning adds over the sensor itself. Both use identical poses and frames.

Strongest simple feed-forward competitor. We run DepthSplat over sliding windows, with its predicted depth affine-aligned per frame to the LiDAR by least squares. We also run a variant with LiDAR concatenated as an extra input channel, fine-tuned on the same ScanNet++ iPhone training split with the same GPU-hours and tuning budget as our model. DepthSplat reports nearly 2 dB PSNR over MVSplat on DL3DV. For this phone-LiDAR setting, no reported result was found on Lacuna for DepthSplat on ScanNet++ iPhone or MuSHRoom. We will run it ourselves.

Strongest streaming feed-forward method. LongSplat is the closest competitor on long sequences. It reports beating DepthSplat by 3.6 dB PSNR on long DL3DV sequences, with 44% fewer splats. We retrain it on the same data. For fairness we give it LiDAR as an input channel, which isolates our measurement-update design from mere sensor access.

Quality ceiling. AGS-Mesh represents per-scene optimization. It reports mesh F-score improving from about 0.60 to 0.91 with 2DGS on MuSHRoom, where MonoSDF reaches 0.8753 and Splatfacto reaches rendering PSNR up to 26.07. We treat it as a ceiling and report its wall-clock time alongside.

Beating them. Our success criteria differ by baseline:

  • Against both feed-forward baselines: better boundary accuracy and lower temporal jitter at matched or better PSNR, at comparable frame rate.
  • Against AGS-Mesh: within a few points of F-score at one to two orders of magnitude less time.

Key-ingredient ablation. The decisive ablation keeps the architecture and inputs but removes map state. The update head is disabled, and each frame spawns splats that are only pruned, as in LongSplat. If this matches the full model, the claim fails. A second ablation removes the uncertainty channel to test whether learned gains matter.

Why not just…?

Why not just align monocular depth to the LiDAR with a per-frame scale and shift, then run an existing feed-forward model? This is the most serious alternative. It will likely remove most global drift, and on whole-image metrics such as AbsRel it may come close to our model. It cannot fix errors that are local and biased, because one scale and shift per frame cannot move an edge. It also cannot use other viewpoints. The aligned-DepthSplat baseline settles the question with three measurements:

  • error binned by distance to depth edges;
  • error binned by range beyond 3 to 4 meters, where AGS-Mesh notes LiDAR degrades;
  • per-point temporal variance.

If our gains vanish in all three bins, the proposal reduces to affine alignment and should be abandoned.

Why not just denoise or upsample each depth frame with a strong 2D network first? Per-frame filtering sees one viewpoint. The PM-Loss study shows that even clean depth maps tear at boundaries once lifted to 3D. Classical guided upsampling, such as Ferstl et al., is iterative and also risks copying texture into geometry. The stateless ablation is exactly this alternative at equal capacity, trained with the same 3D losses, so the comparison is clean.

Why not just fuse the raw LiDAR volumetrically, which already averages over time? Fusion is stable and fast and may be competitive on near-range flat surfaces. Averaging cancels zero-mean noise, as the multi-frame time-of-flight work of Theobalt et al. exploits, but it preserves consistent edge bleeding. It also yields no photorealistic rendering. The fusion baseline and the edge-binned metrics will show where it wins and where it fails.

Why not just optimize per scene as AGS-Mesh does? It will probably remain more accurate. The question is the time-to-quality curve during a live scan, and we report it directly.

Proposed Experiments

Datasets. We train on ScanNet++ iPhone sequences and test on held-out ScanNet++ scenes. We also test zero-shot on MuSHRoom's iPhone captures, with Kinect captures as a cross-sensor generalization check. Both datasets have laser ground truth.

Metrics. Geometry is measured by accuracy, completeness and F-score against the laser surface. Rendered depth is evaluated with AbsRel and boundary F1 at depth edges, the boundary metric used by DVD. Temporal stability is the standard deviation of rendered depth at fixed static 3D points across the sequence. We also report PSNR on held-out frames, which matters given that ScanNet++ reports Nerfacto falling from 25.02 to 17.70 when trained on iPhone rather than DSLR data. Finally, we report frames per second, splat count and peak memory over sequences of several hundred frames.

Decisive checks. We run four.

  1. Map state versus no map state. Does the full model beat the stateless ablation on edge-binned and range-binned error, and on temporal variance? This is the core claim.
  2. Revisiting. On loops where the camera returns to a region from a new angle, does boundary error on earlier-seen edges drop after the revisit? If the map does not improve on re-observation, it is not doing multi-view refinement.
  3. Texture copying. On ScanNet++ surfaces carrying printed text or posters, do flat surfaces stay flat? This tests whether the RGB branch injects false geometry.
  4. Drift. Does accumulated update error grow over 500+ frames? This addresses the forgetting horizon Flash-Mono works around with submaps. A model that needs the same partitioning would still be useful, but the drift would be reported as a limitation.