Free Space That Remembers When: Gaussian Splatting With Partial Depth in Moving Scenes

Depth sensors such as LiDAR and RGB-D cameras give a 3D reconstruction two kinds of evidence. A laser return says "something is here." The empty stretch of ray in front of that return says "nothing is here." The second kind of evidence, called free space, is one of the strongest tools against floaters: stray, half-transparent blobs that appear in mid-air when a reconstruction is fit only to photographs. The trouble is that free-space evidence is only valid if nothing moves. A car parked at time one and gone at time two leaves the same spot marked both occupied and empty. A method that trusts every empty ray will then either push real geometry out of the scene or let the moving object's ghost survive.

We propose to keep a per-region tally over time of how often each patch of space was hit and how often it was passed through. A region that is consistently empty is safe to clear. A region that is consistently hit is solid. A region that is hit at some times and passed through at others is evidence of motion and should be treated differently. That motion signal then gives each camera frame a mask of which pixels are looking at something transient at that moment, so the depth sensor tells the image loss what to ignore. Everything is defined relative to the sensor's own range uncertainty. The same rule, with one setting, should therefore work for a handheld depth camera in a room with people walking around and for a car's LiDAR on a busy street.

Core Claim

EnerGS is the method we build on. It optimizes 3D Gaussian splatting, a scene representation made of many small colored, semi-transparent ellipsoids rendered by projection, inside an energy field. That field divides space into three parts. In occupied space, a robust loss pulls Gaussians onto LiDAR points. In verified free space, a softplus barrier pushes Gaussians out. In unknown space, the photographs alone decide. Gaussian positions are moved only by this geometric field, and color and opacity only by the photometric loss. This is what removes floaters. On KITTI it lowers the leak score from 12.5 for vanilla 3D Gaussian splatting to 9.2, at 16.80 PSNR, and it reaches 24.81 PSNR on Waymo. EnerGS states openly that its free-space carving assumes a static world.

Our claim is that the static assumption can be replaced, not just patched. We add a fourth part to the energy field, conflicted space, assigned from temporal hit and pass-through counts. Inside it the barrier is switched off and the photometric loss is masked at the times when the region is occupied. The result should reach EnerGS-level floater suppression on dynamic sequences without any semantic detector. Because the bookkeeping is purely geometric and measured in sensor-relative units, it should transfer unchanged between indoor RGB-D and outdoor LiDAR.

Where This Sits

Three lines of prior work motivate the gap, and none closes it.

Transient-robust splatting decides what is transient from 2D image error alone.

  • RobustSplat delays densification and bootstraps transient masks from coarse to fine. It also shows that the obvious fix of disabling densification blurs the result. This finding constrains our design: we keep densification and only gate it.
  • 3DGS-HPC classifies image patches.
  • In both, "what moved" is guessed from appearance. We take it from metric ray evidence instead, which does not depend on lighting or texture.

Feed-forward driving models learn whether each element is dynamic.

  • UniSplat predicts a dynamic score for each Gaussian and writes static content into a streaming memory. It admits that a slow pedestrian scored as static becomes a permanent ghost.
  • Our tallies are revisable by construction. A region accumulates conflict as soon as anything passes through it, so an early mistake is undone by later rays. The scores are also not learned on one domain's coordinate frame.

Explicit dynamic-static decomposition separates foreground from background.

  • CDScene splits depth into monocular and multi-view branches on driving voxel grids.
  • Split4D clusters Gaussians by trajectory.
  • Both work from images and stay within one domain. Neither uses free-space evidence from an active sensor.

From the cross-domain literature we borrow one idea. OccAnyScene shows that sizing primitives relative to the camera frustum lets one model span 0.08 m indoor and 0.5 m outdoor voxels. We size our evidence cells and tolerances relative to each sensor's range noise and beam footprint for the same reason. This replaces EnerGS's grid, which is fixed to benchmark bounds.

Research Program

Evidence cells. The evidence structure is a hashed grid whose cell size grows with distance from the sensor, in proportion to that sensor's angular resolution and range noise. For every sweep or depth frame we cast rays and update two counts per cell, each kept with timestamps:

  • $h$, the number of returns that landed in the cell;
  • $p$, the number of rays that passed through it toward a farther return.

Four-way partition. Cells are labeled as follows:

  • static occupied: $h$ large and $p$ near zero;
  • static free: $p$ large and $h$ near zero;
  • conflicted: both counts substantial and separated in time;
  • unknown: neither count substantial.

Hit-then-occluded patterns are not counted as conflict, since occlusion produces no pass-through. The thresholds are expressed as ratios and as multiples of the per-cell range uncertainty. They are tuned once and frozen across all datasets.

Optimization changes to EnerGS. EnerGS's attraction term and barrier apply only in the static parts of the partition. Conflicted cells get neither. Their geometry is left to cross-time photometric consistency, but only at timestamps when the counts say the cell was empty. Each conflicted cell's occupied time intervals are projected into the synchronized camera images. This gives every frame a transient mask grounded in 3D rather than in image error. That mask multiplies the photometric loss and also gates the densification signal, so a moving object can neither spawn Gaussians nor scar the background.

Unknown space. Unknown space keeps EnerGS's weak prior, so photographs can still fill rooftops and the gaps between cameras that LiDAR never measured.

We will study four things:

  • how the partition behaves for slow movers, which produce conflict only at their leading and trailing edges;
  • how sensitive results are to timing offset between camera and LiDAR, since projected masks inherit any desynchronization;
  • how scale-relative cells compare with fixed cells when the same thresholds are moved from TUM-dynamics to Waymo;
  • whether the conflict masks, used as free labels, agree with annotated dynamic boxes on Waymo.

Baselines

Vanilla 3D Gaussian splatting with LiDAR initialization. This is the do-nothing reference. On KITTI it is the 12.5 leak-score anchor that EnerGS reports.

EnerGS as published. This is the strongest existing method on the exact question of floaters under partial depth priors. Its reported figures are 16.80 PSNR and 9.2 leak score on KITTI, and 24.81 PSNR on Waymo. We will rerun it with its released settings on our sequence splits. Because published EnerGS numbers come from its own scene selection, our comparisons will use our reruns, with the published numbers as a sanity check. Beating it means two things on the dynamic-heavy splits: lower leak score and higher static-region PSNR at equal Gaussian budget and iteration count, and on the static splits, no loss beyond noise.

EnerGS with detector or segmentation masking. Points inside dynamic 3D boxes, or inside masked image regions, are removed before the energy field is built. We run it two ways: with ground-truth Waymo boxes as an oracle, and with an off-the-shelf detector. This is the practical competitor, described below.

RobustSplat stacked on EnerGS. The image-derived transient masks of RobustSplat replace our projected masks, and EnerGS's static field is left unchanged. No reported result was found on Lacuna for RobustSplat on KITTI, Waymo or TUM-dynamics, so this is run fresh with matched compute.

Ablations. Three ablations isolate the key ingredient:

  • conflicted cells relabeled as unknown, which keeps the masks but drops the fourth partition;
  • the partition kept but projected masks disabled;
  • fixed-size cells instead of range-relative cells.

Why not just…?

Why not just remove dynamic objects with a 3D detector or 2D video segmentation first? This is the real threat, especially outdoors, where Waymo boxes cover most vehicles and pedestrians. The evidence against relying on it comes from the image side. Split4D reports that tracker-dependent decomposition (SA4D) reaches 0.592 mIoU against its own 0.893 on Multi-Human. PCF-Lift shows 2D foundation masks are inconsistent across views. Detectors also miss things outside their label sets: opened doors, moved chairs, carried bags, and anything indoors. Our answer is empirical. If the ground-truth-box oracle matches our method on Waymo, we will report that the contribution is detector-free operation and indoor transfer, not outdoor quality. TUM-dynamics, which has no box annotations and includes class-agnostic motion, would then carry the claim.

Why not just drop the free-space barrier when the scene moves? Removing the barrier removes the mechanism that suppresses floaters, and the leak score should climb back toward the vanilla 12.5. The "conflicted as unknown" ablation is the measured version of a softer form of this choice.

Why not just use photometric transient masks such as RobustSplat's? Image-error masks catch movers only after they have already distorted the fit. They also cannot correct an energy field whose free space was carved wrongly from the start, which is the specific failure EnerGS names. The stacked RobustSplat-plus-EnerGS baseline tests directly whether 3D-derived masks add anything beyond 2D ones.

Proposed Experiments

Outdoor. We use KITTI and Waymo sequences split by the fraction of LiDAR returns that fall in annotated dynamic boxes. We report PSNR, SSIM and LPIPS on held-out views, scored only on static pixels, along with EnerGS's leak score. We add a geometric hole rate: the fraction of static LiDAR returns from held-out sweeps that lie farther than a range-relative tolerance from the rendered surface. This rate measures true geometry that was wrongly repelled.

Indoor. TUM-dynamics provides RGB-D sequences with people moving through a room. There we report the same static-region view-synthesis metrics and depth error on static pixels. We found no reported reconstruction result on Lacuna for TUM-dynamics. 4RC reports only trajectory error there, and without a value in our file.

Decisive checks. Two results decide the outcome. The first is a degradation curve: as dynamic fraction rises, EnerGS's leak score and hole rate should worsen while ours stay flat, using one frozen threshold setting from TUM-dynamics to Waymo. The second is the comparison against the ground-truth-box oracle, which separates "works without labels and across domains" from "merely matches labels where they exist."