A LiDAR scanner tells you two things. A surface exists where a laser beam returned, and the air between the sensor and that return is empty. Recent driving-scene reconstruction exploits both facts to stop 3D Gaussian Splatting from scattering stray blobs ("floaters") in mid-air. 3D Gaussian Splatting is a renderer that represents a scene as millions of small colored ellipsoids drawn directly to the screen at real-time rates. Both facts, however, are true only at the moment of the scan. A car parked at a curb makes a surface that later becomes empty road, and a pedestrian makes empty air into a surface for a second. Today's best LiDAR-guided method treats every free-space observation as permanent. That assumption breaks on any street with traffic.
We propose to keep LiDAR's emptiness evidence but stamp it with time, and to use the disagreement between scans as the signal that separates static from moving content. A location seen empty at some moments and occupied at others is, by definition, not static background. This yields an annotation-free static/dynamic split that comes directly from the geometric prior, rather than from 2D segmenters or learned deformation networks. It also lets moving objects be rendered as a few rigid clusters, with no neural network evaluated per ellipsoid at render time.
Core Claim
EnerGS turns sparse LiDAR into a precomputed voxel energy field with three regimes. Occupied space pulls Gaussians toward measured surfaces using a robust Welsch loss. Free space pushes them out with a Softplus barrier. Unknown space applies only a weak prior. EnerGS moves Gaussian centers using this geometric energy alone, and blocks photometric gradients from them. It reports 24.81 dB PSNR on Waymo, 16.80 dB on KITTI, a KITTI leak score (percentage of Gaussians in free space) reduced from 12.5 to 9.2, and over 70 FPS. The file states explicitly that its free-space carving assumes a static world, so moving agents corrupt the partition.
Our claim is that a fourth regime fixes this and simultaneously solves the dynamic decomposition problem. We call this regime contested space: voxels that carry both free and occupied evidence at different times.
For each voxel $v$ we count two things across LiDAR sweeps:
- $n_f(v)$, the number of sweeps whose beams passed through $v$;
- $n_o(v)$, the number of sweeps with a return inside $v$.
From these counts we assign each voxel to one regime:
- Static-occupied: $n_o$ is high and $n_f$ is near zero.
- Free: $n_f$ is high and $n_o$ is near zero.
- Contested: both counts are substantial.
- Unknown: neither count is substantial.
Thresholds come from a binomial noise model of LiDAR returns, so that a single spurious return does not flip a voxel. Static Gaussians feel EnerGS's barrier over the union of free and contested space, because anything ever observed empty cannot hold a permanent surface. Dynamic Gaussians may occupy only the time slices in which their voxels were measured occupied.
Research Program
Building the field. From each sweep we extract connected components of contested occupancy. We then associate components across sweeps with rigid registration of their LiDAR points. Each track becomes a rigid cluster with its own object-frame energy field, accumulated from all sweeps registered into that frame. One side of a car seen at one time and the other side seen later then combine into one surface prior.
Rendering cost. This is the design constraint, not the headline. Per-frame rigid poses of clusters are precomputed. Rendering then costs one rigid transform per cluster, in the spirit of Superpoint Gaussian Splatting, which groups Gaussians into rigidly moving "superpoints" to reach 117–227 FPS. Our clusters differ from SP-GS's superpoints in two ways. They are defined by measured geometry, not by learned proximity in a canonical space. They also need no COLMAP initialization, which SP-GS relies on and which degrades under dynamic occlusion.
Optimization. We keep EnerGS's decoupling unchanged for both branches. Gaussian centers follow only the energy field, and appearance follows photometry. This matters because EnerGS identifies photometric gradients as what pulls centers into free space. That pull is exactly what RobustSplat also blames when densification spawns floaters on transient errors.
Disoccluded background. Ground revealed after a parked car leaves is covered by later sweeps, which mark that voxel free above the road and occupied at road level. The background prior therefore fills in automatically.
Learned shape prior for unseen parts. Rigid clusters whose surfaces were never observed get an optional learned completion step. We fit a DeepSDF-style latent shape code, which completes partial geometry by searching a learned shape space, to the accumulated object-frame points. The completed surface is added to that cluster's occupied regime with low weight. We treat this as a separately ablated extension, not the core.
Baselines
Naive reference. This is vanilla 3DGS trained on the same driving sequences. It shows the floater and ghosting failure that motivates everything else.
Strongest existing static-prior method. This is EnerGS run unchanged on the same Waymo and KITTI sequences, with its released hyperparameters and the same iteration budget as ours. Its reported 24.81 dB Waymo PSNR and 16.80 dB KITTI PSNR are image-level numbers. We will rerun it rather than copy them, so that our dynamic-region masks and test frames apply identically. Beating it means three things together:
- higher PSNR and lower LPIPS inside moving-object regions;
- lower leak score in background regions that were once occupied by movers;
- no drop below its whole-image PSNR on largely static sequences.
Strongest competitor on this specific question. This is a box-masked EnerGS that we construct ourselves. We remove LiDAR returns and carved rays inside Waymo's annotated 3D boxes, and model each box as a rigid Gaussian set driven by the ground-truth tracks. It uses human annotations that our method does not, so it is close to an oracle for annotated vehicles. There is no reported result found on Lacuna for this configuration. We will run it with identical Gaussian budgets and training schedules. We expect to match it on annotated vehicles. We expect to beat it on movers outside the annotation taxonomy (doors, carts, unlabeled pedestrians) and on KITTI sequences where tracks are unavailable.
Learned dynamic baselines. These are 4D Gaussian Splatting, which deforms canonical Gaussians with a space-time feature grid and a small network, and SplitGaussian, which splits static and dynamic branches. Both get a LiDAR depth loss added for fairness, so the comparison isolates how the prior is used rather than whether LiDAR is available.
Ablations of the key ingredient.
- Replace contested space with ordinary per-sweep free space. This recovers EnerGS-style permanent carving and tests whether the time-aware counts are what matter.
- Replace rigid clusters with a per-Gaussian deformation network, to measure the frame-rate cost.
- Remove the shape-code completion.
Why not just…?
Why not just mask dynamic objects with SAM2 and run static splatting? Split4D shows that 2D video segmentation drifts under fast motion and contact, and that native 3D decomposition reaches 0.893 mIoU against 0.592 for a tracking-dependent pipeline on Multi-Human (Split4D). Masking also discards the movers themselves, which we must render. We will include SAM2-masked EnerGS as a baseline and report dynamic-region quality, where it should fail by construction.
Why not just use annotated 3D boxes? This is the real threat, and it is why box-masked EnerGS is our strongest baseline rather than a straw man. If our method does not exceed it outside the labeled categories and on unannotated data, the contribution reduces to "annotation-free at equal quality." That is still useful, but weaker, and we will report it as such.
Why not just use a robust photometric loss? RobustNeRF trims the worst half of residuals and treats distractors as outliers. This assumes movers are sparse, and it discards rather than reconstructs them. EnerGS further argues that photometric gradients on centers cause floaters regardless of the loss shape. A trimmed-loss 3DGS with RobustSplat's delayed densification will be run on dense-traffic Waymo segments to test this directly.
Why not just a deformation network like 4D-GS? Monocular Dynamic View Synthesis: A Reality Check shows that unconstrained deformation models lose 4–5 dB on realistic trajectories. Driving captures are one forward-moving rig, close to that regime. The comparison above, at matched LiDAR supervision, decides the question.
Proposed Experiments
We evaluate on Waymo Open Dataset and KITTI, the benchmarks used by EnerGS, with held-out frames.
Metrics. We report PSNR, SSIM, and LPIPS three ways:
- on full images;
- on dynamic-object pixels, with masks derived from projected boxes and used only for evaluation;
- on disoccluded background pixels.
We additionally report leak score split into static and dynamic regions, and rendering FPS on the same GPU as EnerGS.
Decisive checks.
- Contested-space thresholds should recover annotated movers with high recall without any annotation. We measure this by voxel-level agreement with boxes.
- Background leak in voxels once occupied by parked-then-moving cars should fall clearly below EnerGS. This is the specific failure the method targets.
- Frame rate should stay at or above EnerGS's reported 70 FPS. Replacing clusters with a deformation network in the ablation should cost frames, as SP-GS's non-rigid variant halves throughput.
Falsification. If disabling contested space changes nothing on dense-traffic sequences, the core claim is wrong.
References
- EnerGS: Energy-Based Gaussian Splatting with Partial Geometric Priors
- Superpoint Gaussian Splatting for Real-Time High-Fidelity Dynamic Scene Reconstruction
- RobustSplat: Decoupling Densification and Dynamics for Transient-Free 3DGS
- DeepSDF: Learning Continuous Signed Distance Functions for Shape Representation
- 4D Gaussian Splatting for Real-Time Dynamic Scene Rendering
- SplitGaussian: Reconstructing Dynamic Scenes via Visual Geometry Decomposition
- Split4D: Decomposed 4D Scene Reconstruction Without Video Segmentation
- RobustNeRF: Ignoring Distractors with Robust Losses
- Monocular Dynamic View Synthesis: A Reality Check