Time-Indexed Free Space for Dynamic Gaussian Splatting from Sparse Cameras and LiDAR

Gaussian splatting represents a scene as millions of small colored 3D blobs fitted to photographs. It needs a starting cloud of 3D points, usually computed by matching features across many overlapping images. In driving data that starting cloud is poor for two reasons. The cameras are few and point in different directions, so feature matching fails. Cars and pedestrians also move, so the scene stops being consistent across images. LiDAR, a laser scanner that measures distance, can replace the missing starting cloud. A LiDAR ray tells you two things: something exists where the ray stops, and nothing exists along the path before it stops. That second fact, "this space is empty," is the strongest known tool against floaters, the stray blobs that sparse-view optimization parks in front of cameras. Current methods use it only by assuming the world never changes. A car that drives through a voxel once marks it empty forever, and the method then deletes the car.

We propose to index free space by time. A ray fired at time $t$ certifies emptiness only at time $t$. Once emptiness carries a time stamp, the same evidence does two jobs. It purges floaters, as before. It also tells static parts of the scene from moving ones without any learned classifier, because a static blob may never sit in space that was ever observed empty, while a moving blob only has to avoid space observed empty at the moment it is there. A further benefit follows for the third problem, missing geometry. A moving rigid object is seen from many angles over time. If its occupancy and free-space evidence is gathered in the object's own frame, the unobserved backside shrinks without hallucinating anything.

Core Claim

The closest work is EnerGS. It divides space into LiDAR-occupied, LiDAR-free and unknown regions. It pulls blobs toward occupied space with a robust penalty, pushes them out of free space with a smooth repulsive barrier, and leaves unknown space free for photometric gradients to fill. It reports the best PSNR among its baselines on static splits: 16.80 on KITTI and 24.81 on Waymo. It also lowers the "leak" score, the share of blobs placed in true free space, from 12.5 for vanilla 3DGS to 9.2 on KITTI. The research file states plainly that its free-space grid is static, so moving actors corrupt it and the barrier carves holes in them.

Our claim is that the fix is a change in what free-space evidence means, not an added dynamic-object module. Let $F_t$ be the set of voxels crossed by LiDAR rays timestamped near $t$ before they return. Let $\mathcal{S}$ be the static blobs, with fixed centers $x_i$. Let $\mathcal{D}$ be the dynamic blobs, with time-varying centers $x_j(t)$. Let $\phi$ be the softplus barrier from EnerGS, applied to the signed distance into a set. The free-space energy is then

$$E_{\text{free}} = \sum_{i \in \mathcal{S}} \phi\big(d(x_i, \textstyle\bigcup_t F_t)\big) + \sum_{j \in \mathcal{D}} \sum_t \phi\big(d(x_j(t), F_t)\big).$$

The two terms differ in how much evidence they let constrain each kind of blob. A static blob is constrained by the union of all empty-space observations, so it is pushed hard. A dynamic blob is constrained only by observations at its own moment. A blob that the first term would repel but the second would not is, by definition, explained only by motion. That gives a per-blob static/dynamic assignment grounded in ray geometry rather than appearance.

This contrasts with UniSplat. UniSplat predicts a learned dynamic score per blob and admits that a pedestrian misclassified as static is written permanently into its background memory, leaving trailing ghosts. In our formulation, such a misassignment raises the static free-space energy as soon as any later ray passes through the vacated space, so the error can be detected and reversed.

Research Program

The system starts from LiDAR returns rather than image feature matching. This deliberately sidesteps the collapse of matching under wide baselines and sparse views that Large Baseline Matching of Scale Invariant Features and CoMapGS describe. Every point starts as static.

Time-indexed free space is built from per-point LiDAR timestamps. A spinning LiDAR sweep takes about a tenth of a second, so we use per-point timestamps rather than per-sweep ones. We also widen each free-space band by a margin tied to a plausible actor speed, so that the edge of a moving car is not certified empty by a ray fired a few milliseconds earlier.

The dynamic blobs carry a low-capacity motion model. In the main variant this is a per-cluster rigid trajectory, which suits vehicles. A small deformation field is added for articulated actors such as pedestrians. We follow EnerGS in updating blob positions from the energy rather than from photometric loss in LiDAR-covered regions. This is the ingredient the file credits with preventing floaters under sparse views. Photometric gradients keep control in unknown space.

Blobs migrate between the static and dynamic sets on a schedule. When the static energy of a blob stays high while its time-local energy stays low, it becomes dynamic. Dynamic blobs whose fitted motion shrinks to near zero return to the static set.

For completion, each rigid dynamic cluster pools its occupied and free voxels across time after the scene is transformed into the cluster's own frame. Unknown space for the object becomes only what was never seen from any angle across the clip. That remainder stays compliant, as in EnerGS. We do not introduce a learned shape prior in the main method, because the evidence-only version is the cleaner test of the idea. A learned prior such as PCN is tested as an add-on.

Baselines

The naive reference point is a dynamic splatting model fit to the same sparse cameras with LiDAR points as initialization and a plain LiDAR depth loss. It has no free-space reasoning at all. It shows how much free space contributes in the first place.

The strongest per-scene competitor on the floater question is EnerGS run unchanged on dynamic sequences, with its static free-space grid. It is the only method in the file that uses free-space evidence to suppress sparse-view floaters, and its reported Waymo static-split PSNR of 24.81 and KITTI leak score of 9.2 set the bar for that ingredient. We run it on exactly our sequences, cameras and LiDAR, with the same iteration count and the same hyperparameter search budget per method. Beating it means matching its static-region PSNR and leak score while clearly reducing holes on moving actors. We expect static EnerGS to fail on precisely those actors.

The strongest dynamic competitor is UniSplat, a feed-forward model whose reported results are 25.12 dB novel-view PSNR on Waymo and 25.37 dB on nuScenes. Those numbers come from a different protocol, a single forward pass with no per-scene optimization, so we will not compare against them directly. We will evaluate the released model on our held-out views if it is available. We will also report ghost-trail rates alongside PSNR, because its documented failure is persistent ghosts rather than average image quality. GC-4DGS, which reports +1.58 dB over 4DGS on N3DV with three cameras using depth-ranking regularizers, is included as the strongest sparse-camera dynamic method that uses no range sensor. Its comparison is run on the driving data with its depth-ranking losses intact.

Three ablations isolate the key ingredient. The first applies only the time-local term to every blob, with no static/dynamic split. The second uses ground-truth 3D boxes to route LiDAR points and blobs into static and dynamic sets, then runs EnerGS on the static part and a box-tracked model on actors. This is an oracle upper reference. The third removes object-frame aggregation, to measure its effect on backside completeness.

Why not just…?

Why not just use the dataset's 3D bounding boxes to cut moving objects out of the LiDAR and run EnerGS on the rest? This is the most serious threat, because Waymo and KITTI-360 ship tracked boxes. On rigid, labeled vehicles the box-routed oracle may match or beat us, and we will report that honestly. The case for our method rests on three settings the oracle does not cover. The first is unlabeled movers, such as carts, animals and swinging doors. The second is non-rigid actors whose boxes poorly bound them. The third is any deployment without human-annotated boxes. We will report results separately on labeled-rigid, labeled-articulated and unlabeled moving regions. If our method only ties the oracle on rigid cars, its contribution is removing the labels, not improving quality.

Why not just apply a per-frame free-space barrier to a 4D model, without any static/dynamic split? This handles moving actors but gives up the strong union constraint on static blobs. Static floaters seen empty only at some timestamps would then go unpunished at others. The first ablation tests exactly this. If its static-region leak score equals ours, the split adds nothing beyond classification and we would say so.

Why not just let a learned dynamic score decide what is static, as UniSplat does? UniSplat's own analysis reports that one misclassification permanently pollutes its background memory. We will measure ghost trails where an actor previously stood, using box-derived masks only for evaluation. Our prediction is that geometric evidence removes such trails once a later ray crosses the space. Trails will remain where no later ray ever does, and we will quantify that residual.

Why not just complete each frame's points with PCN and initialize 4D splatting from them? The file notes that PCN was designed for isolated, category-constrained objects. It also notes that frame-by-frame completion has no temporal coupling, and that completed points are still dragged into floaters if photometric loss drives positions. We test PCN both as a replacement for object-frame aggregation and as a prior added only to the never-observed remainder. The question is whether learned completion helps after real multi-angle evidence has been used.

Proposed Experiments

We use dynamic segments of Waymo Open and KITTI-360, plus nuScenes for its sparser 32-beam LiDAR. Cameras are thinned to induce the sparse-view regime by keeping one front camera, or a subsampled surround set, and every fourth frame. Held-out frames and cameras are used for evaluation.

Image quality is reported as PSNR, SSIM and LPIPS over the whole image and separately inside dynamic-actor masks. Geometry uses the leak score defined by EnerGS, Chamfer distance against held-out LiDAR sweeps, and an actor hole rate: the fraction of held-out LiDAR returns on moving actors with no blob nearby. For dynamic LiDAR geometry, the reference point is STGC-NeRF's KITTI-360 Chamfer distance of 0.0997, though its NeRF, LiDAR-only protocol differs from ours. Backside completeness is measured on vehicles, scoring the completed blob cloud against LiDAR points observed from the opposite side later in the clip after withholding those sweeps from training.

The decisive check has three parts. Against static EnerGS, the actor hole rate must drop sharply while the static leak score stays at or below EnerGS's. Against the per-frame-only ablation, the static leak score must be lower. Against the learned-score dynamic baseline, ghost trails must be fewer. If the box oracle matches us on rigid cars but we lead on unlabeled and articulated movers, the main claim holds in its labels-free form. If the per-frame-only ablation matches us everywhere, the static/dynamic consistency argument fails and the paper reduces to "time-stamp your free space."