Teaching Feed-Forward Driving Reconstructors Where Space Is Empty

Feed-forward 3D reconstruction networks look at a handful of car-camera images and, in a fraction of a second, output a full 3D scene made of 3D Gaussians: small, fuzzy, semi-transparent blobs that can be rendered from any viewpoint. These networks run without a laser scanner (LiDAR), and they are judged mostly by how good their re-rendered images look. That is a weak test. On large outdoor scenes with LiDAR ground truth, GauU-Scene V2 found that methods with the best image scores can have poor geometry: about two-thirds of the Gaussians in an optimized scene are nearly transparent "floaters" that improve the picture without sitting on any real surface. The driving reconstructors that report steady image-quality gains, DrivingForward, VGD and DrivingScene, have never been checked against LiDAR geometry.

We propose to use LiDAR at training time only, and to use it for what it measures best. A LiDAR return says more than "the surface is at distance d along this ray." It also says that nothing solid lies between the sensor and d. We train a camera-only feed-forward reconstructor on KITTI with a loss on the full profile of where each ray stops inside the predicted Gaussians. Opacity placed in LiDAR-observed empty space is penalized, and the ray is pushed to stop sharply at the return. We then test at inference with no LiDAR at all. The central claim is that the usual way of adding LiDAR, a loss on the rendered depth, is satisfied by a blur of floaters whose average happens to land at the right distance. Supervising free space and stopping location instead should produce Gaussians that sit on surfaces. We expect this to show up in 3D point-cloud error and in renders from viewpoints far from the inputs, even where rendered-depth error looks equal.

Core Claim

Consider a camera ray that passes through a pixel where a visibility-filtered LiDAR point projects at distance $d_r$. The Gaussian rasterizer sorts the Gaussians hit by the ray by depth $t_i$ and gives each one a compositing weight $w_i$, the share of the pixel's final color it contributes. The rendered depth is $\sum_i w_i t_i$. A loss on that sum has many minimizers, including weight spread across floaters in front of and behind the surface. We instead supervise the weights directly:

$$ \mathcal{L}_{\text{term}} = \sum_r \Big[ \sum_{t_i < d_r - \epsilon_r} w_i \;+\; \Big(1 - \sum_{|t_i - d_r| \le \epsilon_r} w_i\Big) \Big], $$

where $\epsilon_r$ is a tolerance band that grows linearly with $d_r$, to absorb LiDAR range noise and the larger size of distant Gaussians. The first term empties the space the laser passed through. The second requires the ray to stop within the band. Nothing at inference changes. The network, its speed and its camera-only inputs stay the same.

Relation to Prior Work

Three feed-forward driving reconstructors motivate the question and supply the architecture. DrivingForward predicts Gaussians from low-overlap surround cameras using self-supervised depth and pose. VGD distills geometry from the VGGT foundation model, a large pretrained network that predicts 3D structure from images, into a compact network. DrivingScene separates the static background from moving objects using a residual-flow network. All three train on photometric losses, and none measures 3D error against LiDAR. We keep their inference path and change only the training signal and the evaluation.

Two papers constrain the design. GauU-Scene V2 shows that image metrics and 3D accuracy come apart, but it studies only per-scene optimization. We carry its question into the feed-forward setting and target its floater finding directly. MonoSKD argues that strict point-to-pixel alignment between LiDAR and camera features breaks down, and uses rank correlation between network features instead, but only for box detection. Our loss acts on geometry rather than features, and KITTI's calibration keeps point projection reliable. We therefore take MonoSKD's warning as a reason to filter projected points for visibility, not as a reason to avoid points.

The closest method is PM-Loss. It fixes the tearing at object edges that appears when depth maps are lifted into Gaussians, by pulling Gaussian centers toward a VGGT-generated pointmap with a Chamfer loss. It needs no LiDAR, it was tested only on indoor and object data, and it inherits the pointmap network's errors on reflective and moving surfaces. PM-Loss constrains where Gaussian centers are, not how opacity is spread along a ray. It is therefore both a competitor and a possible complement. Joint Modeling of Uncertainty removes floaters by down-weighting uncertain Gaussians during per-scene optimization. Our approach amortizes a similar cleanup into training, with a physical signal in place of uncertainty heuristics.

Research Program

We will build a feed-forward Gaussian reconstructor on KITTI's calibrated camera, Velodyne and pose data (KITTI). It will follow DrivingForward's design, reimplemented, with DrivingScene's dynamic branch added as a variant. It takes the left camera at several consecutive timesteps as input. KITTI's stereo right camera is never an input, and it provides held-out views with a known 0.54 m sideways shift.

LiDAR points are projected into the input and supervision views and filtered for occlusion by comparing each point against its neighbours in the projected depth. They then enter $\mathcal{L}_{\text{term}}$ alongside the photometric loss. Rays toward the sky and beyond LiDAR range receive no geometric supervision.

We will also build an evaluation protocol that the driving literature lacks. The first metric is Chamfer distance and F-score at 0.1, 0.2 and 0.5 m, computed between opacity-weighted Gaussian centers and LiDAR sweeps aggregated over the sequence and held out from training. The second is off-surface opacity mass, the fraction of total compositing weight lying in LiDAR-observed free space. We report rendered-depth absolute relative error both from the weighted average and from the median-termination depth. The gap between those two numbers is itself a direct diagnostic of floaters.

Results will be broken down into road surface (identified by fitting a ground plane to the LiDAR), object silhouettes, and distance bins. The breakdown matters because textureless asphalt and object edges are where photometric matching gives the least signal.

Baselines

The naive reference is the same network trained with photometric loss only, which is the regime of DrivingForward. Every baseline is trained on identical KITTI splits and input frames, at the same resolution and schedule, on the same GPU budget. Each gets an equal hyperparameter sweep over its loss weight, eight settings chosen on a validation split. We found no reported result on Lacuna for feed-forward Gaussian reconstruction on KITTI, so all KITTI numbers will come from our own runs.

To tie our reimplementation to published numbers, we also train the photometric and dynamic variants on nuScenes. There we compare against DrivingScene's reported 28.76 dB PSNR, 0.895 SSIM and 0.227 Abs Rel (DrivingScene) and DrivingForward's reported multi-frame 26.06 dB PSNR (DrivingForward). On nuScenes we add our geometric metrics, which neither paper reports.

The strongest competitor on our specific question is LiDAR rendered-depth supervision: an L1 loss between rendered weighted-average depth and projected LiDAR depth. It uses exactly the same data as our method, it is the standard way practitioners would add LiDAR, and it is the ablation that isolates our key ingredient. That ingredient is supervising the stopping profile rather than its average. A second ablation keeps only the free-space term, and a third keeps only the band term, so we can see which half does the work.

The strongest LiDAR-free competitor is PM-Loss with VGGT pointmaps (PM-Loss), run with its published alignment and Chamfer settings. We also test PM-Loss combined with our loss.

Beating the baselines means the following. At matched right-camera PSNR (within 0.3 dB), our method should achieve lower Chamfer distance and at least halve off-surface opacity mass relative to rendered-depth L1. These gains should be largest on road and silhouette regions, with no loss of the sub-second inference time.

Why not just…?

Why not just supervise rendered depth with LiDAR? This is the real threat. If rendered-depth L1 matches our loss on Chamfer distance and floater mass, our central claim is wrong. The contribution would then shrink to the geometric benchmark plus the finding that the simple loss suffices, and we would report exactly that. Our reason to expect a gap is structural. An average-depth loss places no constraint on how weight is spread along the ray, and GauU-Scene V2 shows that optimizers readily fill that freedom with transparent primitives. The medium-versus-mean depth gap and the free-space opacity mass will show directly whether depth-L1 models hide floaters behind correct averages.

Why not just use PM-Loss or VGGT distillation and skip LiDAR entirely? Needing no LiDAR even at training time would be a real advantage. However, PM-Loss's quality is capped by its pointmap generator, which the paper itself flags as fragile on reflective and moving surfaces, and it was never tested outdoors. VGD likewise distills from an image model rather than from a range measurement. If PM-Loss alone matches our method on KITTI, that is a valuable result in favor of sensor-free training. Our experiments settle the question head to head, and the breakdown by region shows where the foundation-model pointmap fails.

Why not just prune low-opacity Gaussians after prediction? Thresholding opacity removes faint floaters, but it does nothing about moderately opaque Gaussians sitting in front of the surface. It also removes the legitimate semi-transparent Gaussians that soften edges. We include post-hoc pruning at several thresholds on the photometric and depth-L1 models as an extra baseline. If pruning recovers our Chamfer gains without hurting PSNR, the training loss is unnecessary.

Why not just optimize each scene per log? Per-scene 3D Gaussian Splatting takes about nine minutes per scene against 0.29 s for feed-forward inference (DrivingForward), and it cannot run online. We use a LiDAR-supervised per-scene fit only as an upper bound on achievable geometry, not as a competitor.

Decisive Checks

The main experiment is the four-way comparison on KITTI held-out sequences: photometric only, plus rendered-depth L1, plus our stopping-profile loss, and plus PM-Loss. It is scored on Chamfer distance, F-score, off-surface mass, both depth-error variants, and PSNR, SSIM and LPIPS on the held-out right camera.

The hypothesis predicts a specific pattern. Depth-L1 and our loss should reach similar average-depth error, but our loss should win clearly on Chamfer distance, floater mass and the median-versus-mean gap.

The second check tests extrapolation. We render at synthetic shifts beyond the stereo baseline, of 1 m and 2 m sideways and 1 m upward, in the spirit of VR-Drive's viewpoint tests (VR-Drive). We score these renders by rendered-depth agreement with LiDAR, reprojected to the shifted camera. Floaters that are hidden near the input views should become visible there.

The third check is transfer. KITTI-trained models are evaluated without fine-tuning on nuScenes geometry. This tests whether the free-space prior generalizes to a different camera rig with little overlap between cameras.

The claim fails if depth-L1 closes the Chamfer and floater-mass gaps under an equal tuning budget.