Time-Indexed Free Space for Mesh Reconstruction of Driving Scenes

A LiDAR sweep tells you two things: where surfaces are, and where they are not, because each laser ray passed through empty air before hitting something. Recent Gaussian-splatting systems use the "not here" signal to delete stray blobs of geometry floating in empty air. Gaussian splatting is a scene model built from millions of small, soft 3D ellipsoids that are rasterized to images very quickly. These systems break as soon as anything moves. A car that drives away leaves behind space that is "empty" at later times. Treating that space as permanently empty erases the car. The standard alternative throws moving objects away as outliers, which leaves holes in the background they covered. We propose a simple rule. Empty-space evidence at any moment rules out static geometry at that location, but it rules out moving geometry only at that moment. Any cell that is seen both occupied and empty must belong to a moving object. With this rule we separate the scene into a static layer and a set of rigidly moving objects, without segmentation masks. We then extract a clean triangle mesh for each layer inside the optimization loop.

This matters for three reasons. Driving simulators, robotics and AR need explicit meshes rather than soft blobs. They need those meshes to render at interactive rates. And the meshes must be complete, even though any single sensor sees only one side of each object. The time-indexed rule speaks to all three. First, it tells us which layer each observation belongs to. Second, it lets each car be extracted once, in its own coordinate frame, from sweeps taken at many times; the car's sides become visible as it passes, which fills in its shape. Third, because extraction happens once per object rather than once per frame, the dynamic scene renders as static meshes moved by rigid poses. That is the ordinary graphics fast path.

Core Claim

Free-space evidence should be indexed by time and applied asymmetrically. It gives a strong barrier against static surfaces everywhere it was ever observed, and a barrier against a moving object's surface only at the time it was observed. Two further ingredients follow. The first is in-the-loop differentiable surface extraction for the static layer and for each object's canonical (object-centered, time-independent) frame. Differentiable surface extraction means turning a continuous shape field into triangles in a way that still lets rendering and geometry errors flow back to the field. The second is a weak shape prior applied only where no evidence exists at any time. We claim these ingredients produce complete, watertight meshes of both background and vehicles, with fewer floaters than static-assumption methods and fewer holes than outlier-rejection methods, at display frame rates.

The closest work falls into three groups, each missing one piece. EnerGS motivates the proposal and supplies the method. It divides space into occupied, free and unknown regions from partial LiDAR and uses a Boltzmann barrier to push Gaussians out of free space. It reports more than 70 FPS on Waymo. However, it states outright that moving agents violate its free-space assumption, and it outputs Gaussians, not meshes. MILo supplies our extraction machinery. It builds a Delaunay tetrahedral grid from points attached to each Gaussian, learns signed-distance values on it, and runs differentiable Marching Tetrahedra (from DMTet) during training. It obtains meshes with about 4.36M vertices versus 16.39M for post-hoc 2DGS meshing, but it is strictly static. DG-Mesh is the nearest dynamic-mesh system. It deforms canonical Gaussians over time and extracts a mesh at every time step with a differentiable Poisson solver (Shape As Points) followed by Marching Cubes. It needs pre-segmented foregrounds, runs at about 10.9 FPS, and does nothing about unseen back surfaces.

Our proposal differs from all three. We derive the static/dynamic split from contradictions in occupancy over time, not from masks. We extract per object once, not per frame. And we treat completion as a consequence of aggregating evidence across time, with a learned prior only as a last resort.

Research Program

Occupancy evidence. We ray-cast every LiDAR sweep into a sparse voxel grid and record, for each cell $c$ and time $t$, whether it was observed occupied, observed free, or unobserved. The static layer's signed distance field $s(x)$ receives a barrier wherever a cell was ever observed free. It receives an attraction term only where a cell was observed occupied at every time it was observed at all.

Each moving object $k$ has a canonical field $d_k(y)$ and rigid poses $T_k(t)$, which place its canonical frame in the world at each time. Its field receives barrier and attraction terms only at the time of each observation. Written out, the geometric energy for one object is

$$E_k = \sum_{t}\sum_{c \in \mathcal{F}_t} \phi\big(d_k(T_k(t)^{-1}x_c)\big) + \sum_{t}\sum_{c \in \mathcal{O}^{dyn}_t} \rho\big(d_k(T_k(t)^{-1}x_c)\big),$$

where $\mathcal{F}_t$ and $\mathcal{O}^{dyn}_t$ are the cells observed free and observed occupied at time $t$. Cells in $\mathcal{O}^{dyn}_t$ are occupied cells that were observed free at some other time. $x_c$ is the cell center, $\phi$ is the EnerGS-style barrier, and $\rho$ is a robust Welsch penalty. The static energy is the same construction with the time index removed from the free set.

Cells in $\mathcal{O}^{dyn}_t$ are assigned to objects in two steps. Detector tracks give initial poses and assignments. Leftover contradicting cells are clustered to catch objects the detector missed, such as a parked car that starts moving.

Extraction and rendering. Both the static field and each canonical object field live on MILo-style Delaunay grids built around their own Gaussians. Differentiable Marching Tetrahedra runs in the loop, with MILo's two-way consistency between rendered Gaussian and mesh depth and normals. Poses $T_k(t)$ are refined jointly, which spreads each car's observations over its whole canonical surface. After training, each vehicle is a single fixed mesh with a neural texture, rendered by hardware rasterization in the manner of Real-Time Neural Rasterization for Large Scenes. Only the poses change per frame, so rendering cost should not grow with the number of time steps.

Residual completion. Some regions are unobserved at every time, such as the far side of a car seen from one lane. There we add a weak signed-distance prior from a one-step completion of the aggregated canonical points by 3D MeanFlow, which completes a shape in about 5 ms. The prior is gated off wherever any observation exists. This avoids the flicker that 3D Dynamic Point Cloud Inpainting documents when completion is applied independently per frame, because completion here happens once per object in its canonical frame.

Scope. We limit the dynamic layer to rigid agents. Pedestrians and other deforming bodies stay in a Gaussian-only dynamic residual without mesh guarantees. The canonical-deformation line (D-NeRF, DG-Mesh) shows that non-rigid and topology-changing motion is a separate hard problem, and we do not claim to solve it.

Baselines

The naive reference is vanilla 3DGS with post-hoc TSDF meshing on the same Waymo and KITTI sequences. It shows how bad floaters and erased vehicles are when nothing is done.

The strongest static competitor is EnerGS, run with its published energy terms on our exact splits, iteration budget and GPU. EnerGS reports 24.81 dB PSNR on Waymo, 16.80 dB on KITTI, and a KITTI leak ratio of 9.2 against 12.5 for vanilla 3DGS (EnerGS). We re-run it rather than copy these numbers, because our scenes are selected to contain moving agents. Beating it means lower leak ratio and higher PSNR on frames with moving vehicles, while staying within 0.3 dB on static-only frames.

The strongest distractor-handling competitor is RobustSplat followed by MILo-style meshing. No reported result found on Lacuna for RobustSplat on Waymo or KITTI. Beating it means smaller background holes in regions that transient vehicles occluded.

For dynamic meshing, DG-Mesh is run on per-vehicle crops using ground-truth box masks. This gives it an advantage it would not have in deployment. It reports about 10.9 FPS, and Chamfer distance roughly 50% below 4D-GS on its synthetic set, but no reported result found on Lacuna for DG-Mesh on driving data.

The key ablation replaces the time-indexed rule with EnerGS's time-free partition, keeping everything else fixed. A second ablation removes the MeanFlow prior. A third replaces contradiction-based assignment with detector boxes alone. Every method gets the same LiDAR, cameras, tuning sweep of five configurations, and a single-GPU budget.

Why not just…?

Why not just mask detected vehicles out, run EnerGS or MILo on the static rest, and Poisson-mesh the accumulated LiDAR inside each tracked box? This is the real threat. On Waymo, with good tracks, it may match us. We will run exactly this pipeline, fed by both ground-truth and detector boxes. We expect the gap to appear in three places. One is objects the detector misses or that change state, such as parked cars pulling out. Another is box jitter, which smears the aggregated points and which post-hoc Poisson cannot correct because poses are not refined against images. The third is thin structures, where MILo shows post-hoc meshing erodes or inflates surfaces. If the gap is negligible even with detector boxes, the contribution shrinks to a cheaper engineering pipeline, and we will report that honestly.

Why not just fit CAD car templates? Templates are complete and fast. But DG-Mesh documents template failure whenever shape departs from the template. Driving scenes also contain trailers, buses, open doors and roof loads. We will report template fitting on vehicle Chamfer distance, broken down by how typical each vehicle is.

Why not just run 3D MeanFlow on each frame's partial scan? Per-frame completion flickers, as the dynamic inpainting work shows. It also ignores the fact that later sweeps actually observe the missing side. Our no-prior ablation versus per-frame completion measures both effects.

Proposed Experiments

We use Waymo Open and KITTI sequences that contain moving vehicles, each split into training and held-out camera frames and LiDAR sweeps. We measure image quality with PSNR, SSIM and LPIPS on held-out frames, reported separately for moving-vehicle pixels. We measure floaters with the EnerGS leak ratio. We measure geometry with Chamfer distance to held-out LiDAR returns, split into static and dynamic parts. We measure dynamic erasure as the fraction of held-out returns on moving agents with no surface within 20 cm. Latency is FPS at 1080p.

For completion, we withhold all sweeps from one side of selected vehicle tracks and measure Chamfer distance on the hidden side.

The decisive check is the time-indexed-versus-time-free ablation on moving-vehicle regions. If dynamic erasure does not fall sharply with the time-indexed rule, the core claim is wrong. The second check is whether the masked-pipeline baseline closes the gap under detector boxes. The third is whether mesh rendering stays at or above 60 FPS as sequence length doubles, where DG-Mesh-style per-frame extraction cannot.