Reconstructing a driving scene so it can be re-rendered from new viewpoints means separating what stays put (roads, buildings, parked cars) from what moves (traffic, pedestrians). Current reconstruction methods for driving logs make this split by category. A 2D image segmenter labels pixels as "building" or "vehicle," and that label decides whether a 3D primitive is frozen in place or allowed to move. Category is a poor stand-in for motion. A parked car is a vehicle that never moves. A rare truck the segmenter calls a building is a moving object that gets frozen into the background as a permanent ghost.
We propose to make the static/dynamic decision from measured motion instead. The motion comes from a camera–LiDAR scene-flow network, a model that predicts how every 3D point moves between two frames. It is trained only on synthetic data, and semantic labels enter only as a weak prior. The bet rests on a pattern already visible in the literature. Motion estimators trained purely on synthetic scenes transfer to real driving data without fine-tuning, while semantic segmenters fail on exactly the rare cases that matter. Motion is a geometric quantity that obeys the same rules in simulation and reality. Category membership is not.
Core Claim
The reconstruction we build on is 3D Gaussian Splatting, which represents a scene as a cloud of small colored, semi-transparent ellipsoids rendered by projection. Our claim is that in a driving 4D Gaussian reconstruction, gating each Gaussian between "frozen" and "deformable" using accumulated, synthetic-trained cross-modal flow evidence will remove more ghosting on misclassified movers than semantic gating does. It will also keep parked objects sharp, without giving up background quality. A second, smaller claim concerns the projection step. Because cameras and spinning LiDARs do not sample a moving object at the same instant, the same flow should be used to shift LiDAR points to each camera's capture time before image features or depth are attached to them.
The closest work is Urban4D. It splits Gaussians into "definite static" and "potentially dynamic" using SegFormer labels, enforces static invariance on the former, and attaches a time-conditioned deformation network to the latter. Its authors acknowledge that a misclassified moving object gets frozen into the background. We keep its deformation machinery but replace the decision rule.
Split4D shows that tying identity to 3D motion trajectories beats relying on 2D video trackers: 0.893 against 0.592 mean intersection-over-union for SA4D on Multi-Human. That supports motion as the "glue." However, Split4D is vision-only, is optimized per scene, and learns motion from scratch rather than importing a transferable motion prior. We read it as motivation, not as a competitor on driving data.
Research Program
The motion source is CamLiFlow, which passes information back and forth between an image branch and a LiDAR-point branch to jointly estimate 2D optical flow and 3D scene flow. It is trained on synthetic FlyingThings3D and evaluated on real KITTI, where it reports a 4.43% outlier rate. HPLFlowNet gives independent evidence that purely geometric flow transfers from synthetic to real: it is trained on FlyingThings3D and reaches 0.1169 m 3D end-point error on KITTI zero-shot. That makes it the natural LiDAR-only ablation. Both papers supply the method. We reuse their networks frozen and do not propose a new flow architecture.
For each pair of adjacent frames in a driving log, we run the flow network after removing ego-motion. The result is a residual 3D velocity per LiDAR point, plus optical flow for image regions where LiDAR is too sparse, which matters most at long range.
Each Gaussian carries a motion log-odds $\ell_g$. It is initialized from the semantic prior, $\ell_g^{(0)} = \lambda \log \frac{p_{\text{sem}}(g)}{1-p_{\text{sem}}(g)}$, where $p_{\text{sem}}(g)$ is the segmenter's "movable class" probability at the Gaussian's projection and $\lambda<1$ deliberately weakens it. The log-odds is then updated every frame from the flow residual of nearby LiDAR points or projected pixels:
$$ \ell_g^{(t)} = \ell_g^{(t-1)} + \log \frac{p(\lVert \mathbf{v}_g^{(t)} \rVert \mid \text{moving})}{p(\lVert \mathbf{v}_g^{(t)} \rVert \mid \text{static})}, $$
Here $\mathbf{v}_g^{(t)}$ is the ego-compensated flow velocity assigned to Gaussian $g$ at frame $t$. The two likelihoods are fitted once on synthetic flow error statistics and then held fixed. Accumulating evidence over time is what lets a single noisy flow frame be outvoted.
Gaussians whose log-odds stays below a threshold receive Urban4D-style static invariance. The rest get the deformation network. Gating is re-evaluated at a few points during optimization, coordinated with delayed densification in the style of RobustSplat. RobustSplat showed that early densification is what spawns transient Gaussians, so we hold densification until the motion field has settled.
The second component is time-aware lifting. When LiDAR depth or point features are attached to camera pixels, each point is advanced by its estimated velocity times the offset between its firing time and the camera's exposure time. This addresses the misalignment described for classical projection-based fusion such as PointFusion and MVX-Net, which assume synchronized sensors. LiDAR depth then supervises the rendered depth of the Gaussians. This also closes Urban4D's lack of metric range fusion.
The analysis question running through the work is where the sim-to-real boundary bites. We will check whether flow errors on real data concentrate in places synthetic FlyingThings3D lacks, such as wet roads, spray, and very distant objects. We will also check whether the temporal accumulation absorbs those errors.
Baselines
The naive reference is static 3D Gaussian Splatting on the same logs with no dynamic handling. It sets the floor for ghosting and shows how much of the problem is real.
The strongest existing method is Urban4D, which reports 26.92 PSNR on nuScenes overall (against 26.23 for PVG) and 27.61 PSNR on non-sky regions. Any fair comparison must also give Urban4D the LiDAR depth supervision we use. Otherwise a gain could just be LiDAR. So we run "Urban4D + LiDAR depth" with the same Gaussian budget, iteration count, and densification schedule, and tune both on the same held-out scenes. Beating it means three things together. First, no loss in overall PSNR. Second, a clear gain on moving-object regions and on the misclassified-mover subset described below. Third, fewer ghost Gaussians. Whole-image PSNR alone is too diluted by background to be decisive. On dynamic-region PSNR specifically, we found no reported result on Lacuna for nuScenes, Waymo, or PandaSet, so we will compute it ourselves for all methods.
The most dangerous baseline is non-learned geometric motion detection. LiDAR sweeps are registered with ego-motion, and a point is called moving if it has no nearby counterpart in the registered previous sweep. This is plugged into exactly our log-odds gate. It isolates whether the synthetic-trained flow network contributes anything beyond "use motion instead of class."
The ablations isolate each ingredient. We swap CamLiFlow for LiDAR-only HPLFlowNet to test the camera half of the fusion. We remove time-aware lifting. We set $\lambda=0$ (motion only) and $\lambda$ large (semantics dominate). We fine-tune the flow network on KITTI to measure the residual sim-to-real gap. As a secondary geometry reference on KITTI-360, STGC-NeRF reports a Chamfer distance of 0.0997, but it is a LiDAR-only NeRF, so the comparison is indicative rather than head-to-head.
Why not just…?
Why not just use a better segmenter, or SAM2-style masks, in Urban4D? A better segmenter reduces misclassification of rare vehicles but cannot fix the parked-versus-moving confusion. Class does not encode motion, so a perfect segmenter still lets parked cars deform and blur. The stratified evaluation below reports parked and moving objects separately, so a strong-segmenter Urban4D run will show directly how much of our gain survives better semantics. UniLiPs lifts several foundation models into 3D and still relies on motion cues and temporal re-weighting to find static-world violations. That suggests semantics alone are not considered sufficient even by heavy-foundation-model pipelines.
Why not just detect motion by differencing ego-registered LiDAR sweeps, with no learned flow at all? This is a real threat, and we run it as a full baseline. Our expectation that learned flow wins rests on long range and thin objects. There, LiDAR returns are too sparse for differencing, and the camera branch supplies motion evidence. Differencing also fires on occlusion boundaries, where parallax looks like motion. If differencing matches us on the moving and misclassified subsets, the honest conclusion is that the contribution is the motion-gated decomposition itself, not synthetic-to-real flow, and we will report it that way.
Why not just make every Gaussian deformable and let optimization sort it out? Urban4D motivates its split precisely because full 4D deformation lets static backgrounds drift and blur. Our "4DGS everywhere" run with the same deformation network measures that drift on static regions. Gating should match static 3DGS there.
Why not just mask transients and reconstruct only the background, as RobustSplat does? RobustSplat discards dynamics by design, and its authors leave illumination effects unmodeled. Driving simulation needs the moving agents rendered, so this answers a different question. It is still a useful background-quality reference.
Proposed Experiments
The main testbed is nuScenes, chosen for its multi-camera and LiDAR rig and Urban4D's reported numbers. KITTI-360 serves for geometry, and Waymo and PandaSet test generalization.
From the 3D box tracks we compute per-object speed and form three strata. The first is moving objects. The second is stationary objects of movable classes (parked cars). The third is the decisive misclassified-mover subset: tracked objects moving above a speed threshold whose pixels the segmenter labels as a static class.
Metrics follow the requirements above. We report PSNR, SSIM, and LPIPS per stratum and on background. We report a ghost score: rendering residual in regions an object has vacated, measured on frames after it leaves. We report rendered-depth error and Chamfer distance against held-out LiDAR. We also check that flow-network transfer is consistent with CamLiFlow's reported 4.43% outlier rate on KITTI Scene Flow.
The decisive checks are three. On misclassified movers and vacated regions, motion gating should cut the ghost score relative to Urban4D + LiDAR depth while overall PSNR stays at or above that baseline. On parked cars, static gating should give sharper renders than semantic gating.
For time-aware lifting, we inject known timing offsets between camera and LiDAR. Rendered-depth error on fast movers should grow with offset without compensation and stay flat with it.
Finally, the gap between FlyingThings3D-only and KITTI-fine-tuned flow measures how much synthetic-to-real transfer actually limits the system. A small gap would support the thesis that motion is the domain-invariant signal worth importing from simulation.
References
- UrbanGS: Semantic-Guided Gaussian Splatting for Urban Scene Reconstruction
- Split4D: Decomposed 4D Scene Reconstruction Without Video Segmentation
- CamLiFlow: Bidirectional Camera-LiDAR Fusion for Joint Optical Flow and Scene Flow Estimation
- HPLFlowNet: Hierarchical Permutohedral Lattice FlowNet for Scene Flow Estimation on Large-Scale Point Clouds
- RobustSplat: Decoupling Densification and Dynamics for Transient-Free 3DGS
- PointFusion: Deep Sensor Fusion for 3D Bounding Box Estimation
- MVX-Net: Multimodal VoxelNet for 3D Object Detection
- STGC-NeRF: Spatial-Temporal Geometric Consistency for LiDAR Neural Radiance Fields in Dynamic Scenes
- UniLiPs: Unified LiDAR Pseudo-Labeling with Geometry-Grounded Dynamic Scene Decomposition