A spinning LiDAR does more than report where surfaces are. Every beam that returns from 40 meters away also shows that the space between the sensor and 40 meters was empty at that instant. When a car occupies a spot in one sweep and a later beam passes cleanly through that spot, the sensor has directly observed that the car moved. We propose to hand this "the beam went through it" evidence to a network that rebuilds a dynamic driving scene in one forward pass. The network would take a few seconds of LiDAR sweeps and output 3D Gaussians, small soft ellipsoid primitives that can be rendered quickly. From those Gaussians one can synthesize new LiDAR scans at unseen poses and times, with no per-scene optimization. Today this is done by fitting each scene for hours, as in LiDAR4D, or about 1.5 hours with GS-LiDAR.
The trick that makes the evidence cheap is the spherical range image. It unwraps each sweep into a 2D grid, with rows for beam elevation, columns for azimuth, and pixel values for measured distance. If one sweep's points are warped into another sweep's grid, checking whether a beam passed beyond a previously seen surface becomes a per-pixel subtraction. No 3D neighbor search is needed. In two sentences: warp every past and future sweep into the current sweep's range image and compare distances pixel by pixel, which tells the network for free which surfaces were vacated and which were merely hidden. A feed-forward model given that signal should separate static background from moving objects and fill in occluded background without external segmentation, and do it far faster than 3D cross-time queries.
Core Claim
Explicit free-space residuals computed in range-image coordinates are an informative input for feed-forward dynamic reconstruction. A range-image network that receives them will do three things. It will beat an otherwise identical network that sees only stacked multi-sweep range images. It will beat a pipeline that first removes moving points with a segmentation model and then reconstructs statically. And it will approach per-scene optimized panoramic Gaussian splatting on KITTI-360 dynamic LiDAR view synthesis while running in well under a second.
The central quantity is simple. Let $r_j(u,v)$ be the measured range at pixel $(u,v)$ of target sweep $j$. Let $\hat r_{i\to j}(u,v)$ be the range of the nearest point from sweep $i$ after transforming it by the known ego-poses into $j$'s spherical grid. The free-space residual is
$$ e_{i\to j}(u,v) = r_j(u,v) - \hat r_{i\to j}(u,v). $$
A large positive $e$ means sweep $j$'s beam travelled past a surface that sweep $i$ saw. That surface was therefore vacated, which is evidence of motion. A negative $e$ means sweep $i$'s surface is hidden behind something in sweep $j$. That is occlusion, not evidence of motion. Near-zero $e$ is consistent with a static surface.
This sign asymmetry is what RGB-based feed-forward methods lack. Photometric inconsistency cannot tell "moved" from "occluded", and DGGS needs a pretrained segmenter plus an auxiliary loss to make exactly that distinction.
Where This Sits
The proposal draws on three lines of work, each in a different role.
The first supplies methods. DELFlow estimates scene flow by keeping exact $(x,y,z)$ coordinates inside range-image pixels, searching neighbors in 2D windows, and warping points between frames through a "warping index" that resolves pixel collisions. It reports a 25x speedup (52.9ms versus 1324ms). We reuse the warping index to compute $e$ and to gather cross-sweep features. DELFlow only estimates motion and does not reconstruct or render; we add both. GS-LiDAR renders LiDAR scans from 2D disk Gaussians using exact intersections between sensor rays and splats. Its ablation shows the pinhole-style approximation drops PSNR from 28.4 to 21.5. We adopt its renderer unchanged. GS-LiDAR itself is per-scene optimized and models motion as sine-wave vibration; we replace both with a learned feed-forward predictor. CRISP uses cylindrical sliding windows so 2D convolutions wrap seamlessly across the 360-degree seam, and we use the same circular treatment in our backbone.
The second line motivates the work. LGS is the closest feed-forward driving reconstructor. It reports 28.04 PSNR on Waymo camera views and raises dynamic-region PSNR on PandaSet from 17.65 (STORM) to 22.43. It handles moving objects with a Cross-Time Point Query that searches Euclidean neighbors across timestamps, a step its authors flag as poorly scaling. LGS targets camera view synthesis and never uses free-space evidence. We target LiDAR synthesis, index time through range grids, and feed the network an explicit moved-versus-occluded signal. BTimer shows that conditioning a feed-forward Gaussian predictor on a target "bullet time" can freeze a dynamic scene into a static snapshot in 150ms. We borrow that time conditioning for dynamic Gaussians, but BTimer is monocular RGB with admittedly inaccurate geometry. Any4D and D4RT do fast feed-forward 4D reconstruction from video, but they anchor motion to the first frame (Any4D) or decode points on demand rather than producing a renderable field (D4RT). Neither uses sensor geometry.
The third line constrains the design. DGGS shows that pruning distractor Gaussians leaves holes when background is hidden in every input view. We therefore treat hole rate as a primary measurement, not a footnote. LiDAR-NeRF shows that a static-world assumption fails on dynamic LiDAR scenes. The range-image literature (SemanticKITTI, CRISP) warns that a single-layer range image keeps only the first hit per beam. Our design relies on that property, since first-hit semantics is exactly what makes $e$ meaningful. But it means dynamic objects that are thin or partly transparent to the beam will be missed.
Research Program
Inputs and geometry. The model takes a window of $K$ sweeps, starting with $K=5$ spanning roughly half a second, together with their ego-poses from the dataset. Each sweep is projected to a range image with range, intensity and $(x,y,z)$ channels. For every ordered pair $(i,j)$ we warp sweep $i$ into sweep $j$'s grid with the DELFlow-style warping index and compute the residual map $e_{i\to j}$. We also record the warped source features, so cross-time aggregation is a gather through precomputed indices rather than a 3D neighbor search. The residuals are clipped and passed through a soft tolerance that scales with range and beam spacing, because angular quantization makes range noise grow with distance.
Network. A 2D encoder–decoder with circular padding in azimuth processes each target sweep's own channels concatenated with its $K-1$ residual maps and gathered features. Per pixel it predicts disk-Gaussian parameters (offset along the beam, scale, orientation, opacity, intensity, ray-drop logit) and a static/dynamic gate. Following LGS's lesson that one Gaussian per point is too rigid, each pixel may emit a small fixed number of candidate Gaussians with learned opacities. This is cruder than LGS's intervention-trained densify policy, and we compare against it directly.
Static Gaussians from all sweeps are merged in the world frame, with a range-grid non-maximum suppression step to avoid duplicates. Dynamic Gaussians are regressed per target time using a BTimer-style time embedding. Neither the static nor the dynamic branch needs motion labels.
Supervision. Training is self-supervised with held-out sweeps. Given the window, the model renders held-out sweeps inside the window and nearby, using GS-LiDAR's exact ray–splat renderer. Losses are L1 on range, L1 on intensity, and binary cross-entropy on ray drop. The only extra term is a weak prior pushing the gate toward "static" where all residuals are near zero. We deliberately provide no segmentation labels. That way any static/dynamic separation the model learns can be credited to the residual signal and not to annotations.
Training data. We train across many scenes from Waymo Open, nuScenes and PandaSet, and KITTI-360's training sequences. We evaluate on held-out KITTI-360 and nuScenes sequences using the static and dynamic splits of LiDAR4D and GS-LiDAR, plus a cross-dataset transfer split.
Analysis. Beyond headline quality, we study three failure sources the method itself introduces. First, pose error, which manufactures false residuals; we inject controlled pose noise. Second, the spinning sensor's rolling-shutter distortion. Third, the first-hit limitation on thin or partly transmissive objects.
Baselines
The do-nothing reference is static accumulation. Transform all $K$ sweeps into the target pose, keep the nearest return per pixel, and re-project, with no learning. This is what the static assumption buys for free. It should look acceptable on static regions and leave ghost trails on moving objects. Any learned model that fails to beat it clearly in dynamic regions has not learned anything useful about motion. It needs no tuning.
The quality ceiling is per-scene GS-LiDAR, run with its released protocol for its reported roughly 1.5 hours per scene. We also run LiDAR4D where compute allows. The file reports only relative numbers for GS-LiDAR on KITTI-360: a 10.7% depth RMSE reduction and 5.3% Chamfer reduction over LiDAR4D on static scenes, and an 11.5% depth RMSE reduction on dynamic scenes, plus 13.1% on nuScenes. No absolute depth RMSE or Chamfer value for GS-LiDAR was found on Lacuna. The only absolute KITTI-360 scene-level figure in the file is LiDAR-NeRF's Chamfer distance of 0.081 and F-score of 92.49%, an older static-method result. We will therefore rerun GS-LiDAR ourselves on identical sequences and splits. Our success criterion is not to beat this ceiling. It is to come within 15% of its depth RMSE and Chamfer distance on dynamic sequences while being at least three orders of magnitude faster per scene, measured as wall-clock from raw sweeps to renderable scene on one GPU.
The strongest feed-forward competitor is an LGS-style model adapted to LiDAR targets. It uses the same backbone capacity, the same GS-LiDAR renderer and losses, and the same training data and step budget. Temporal aggregation goes through Cross-Time Point Query with 3D nearest-neighbor search, and density comes from a learned densify map, following LGS. LGS reports 28.04 PSNR on Waymo camera views and 22.43 dynamic-region PSNR on PandaSet. These numbers are not comparable to LiDAR metrics, so we reimplement it rather than quote it. Beating it means lower dynamic-region depth RMSE and Chamfer distance at equal compute, and lower latency as point density grows.
The key ablation is the same network with the residual maps removed. It still receives the raw warped sweeps and gathered features, so in principle it could compute the residuals itself. This isolates whether the explicit free-space signal matters or whether the backbone learns it anyway. A second ablation feeds the unsigned magnitude $|e|$, which tests whether the moved-versus-occluded sign is the active ingredient.
Why not just…?
Why not just segment moving points first and then reconstruct statically? One could run a LiDAR moving-object segmenter, delete moving points, and train a static feed-forward reconstructor. This is the most serious threat. The research file cites evidence that 2D-mask-first pipelines propagate label errors into 3D (Overcoming the Static Scene Assumption), and that moving classes are hard for simple segmenters, often below 20% mIoU on SemanticKITTI. But LiDAR motion segmenters that themselves use multi-sweep residuals may do much better than those figures. We will run the strongest segment-then-reconstruct pipeline we can build, with the segmenter trained on SemanticKITTI moving labels and fed the same $K$ sweeps. If it matches our model on static regions, we accept that the gain is confined to dynamic content. The claim survives only if our model wins on dynamic regions, where the deleted objects must still be rendered, and on hole rate.
Why not just stack sweeps and let the network learn free space? This is the key ablation above. If the no-residual variant matches the full model at equal training budget, the core claim is false. The contribution would then shrink to an efficient range-grid temporal gather, and we would report it as such. We expect the explicit signal to matter most with limited data and under cross-dataset transfer, where learned implicit comparisons tend to overfit sensor specifics. We test both conditions.
Why not just use 3D sparse convolutions or point networks instead of range images? They avoid collision loss. But DELFlow measured a 25x latency penalty for comparable geometry without 2D grouping, and computing free-space residuals in 3D requires explicit ray casting through a voxel grid. We include a sparse-voxel variant of the backbone, with residuals computed by voxel ray traversal, and report quality and latency together.
Why not just cut the panorama into cubemap faces and use a pinhole feed-forward splatter? GS-LiDAR's ablation (28.4 to 21.5 PSNR without exact ray–splat intersection) and CRISP's seam artifacts suggest this degrades. We run a cubemap pixelSplat-style variant once to confirm the size of the gap, not to tune it.
Proposed Experiments
The main evaluation is KITTI-360 LiDAR novel space-time view synthesis on the dynamic sequences used by LiDAR4D and GS-LiDAR, plus nuScenes. Metrics are depth RMSE, Chamfer distance, F-score, intensity error, and ray-drop accuracy. Every metric is reported separately for static and dynamic regions, with dynamic regions defined by dataset instance labels on KITTI-360.
Three checks decide the proposal. First, the residual ablation must show a clear dynamic-region gain; if it does not, the core claim fails. Second, on the segment-then-reconstruct comparison, we report ghosting and hole rate. Ghosting is the fraction of rendered returns lying in space that ground truth shows as empty. Hole rate is the fraction of ground-truth static returns with no rendered return within tolerance. Our model must reduce both without being given labels. Third, a latency-scaling curve compares range-grid gathering with 3D cross-time queries as points per sweep grow.
Supporting tests follow. We measure robustness to injected pose noise, since this is where the method should degrade first, and state the noise level at which its advantage disappears. We run cross-dataset transfer by training without KITTI-360 and testing on it. Finally, we check static/dynamic separation by scoring the learned gate against SemanticKITTI moving-point labels, purely as a diagnostic.
References
- LiDAR4D: Dynamic Neural Fields for Novel Space-time View LiDAR Synthesis
- GS-LiDAR: Generating Realistic LiDAR Point Clouds with Panoramic Gaussian Splatting
- Distractor-free Generalizable 3D Gaussian Splatting
- DELFlow: Dense Efficient Learning of Scene Flow for Large-Scale Point Clouds
- CRISP: Cylindrical Rendering for In-Stream Point Clouds
- Learning Gaussian Structure: Intervention-Guided Density Control for Feed-Forward Driving Reconstruction
- Feed-Forward Bullet-Time Reconstruction of Dynamic Scenes from Monocular Videos
- Any4D: Unified Feed-Forward Metric 4D Reconstruction
- Efficiently Reconstructing Dynamic Scenes One D4RT at a Time
- LiDAR-NeRF: Novel LiDAR View Synthesis via Neural Radiance Fields
- SemanticKITTI: A Dataset for Semantic Scene Understanding of LiDAR Sequences