Sweep-Aware Scene Flow: Training LiDAR Motion Estimators in Simulation for Real Sensors

A spinning LiDAR does not take a snapshot. It sweeps a fan of laser beams around the vehicle, so a scan that looks like one 3D picture is really a surface in space and time. Points on one side of the car are measured up to a tenth of a second after points on the other side, and faster rotation buys more scans per second at the cost of fewer points per scan. Synthetic training data for 3D motion estimation ignores all of this. Datasets like FlyingThings3D give a network instantaneous, complete point clouds with perfect motion labels. Networks trained this way still fail on a fifth of real points: RMS-FlowNet, trained only on FlyingThings3D, leaves a 20.3% outlier rate on real KITTI data. We propose that part of this gap is a mismatch in timing and sampling, not only in appearance. If so, it can be closed by simulating the sweep and building the motion estimator around it.

The concrete proposal has two parts. The first is a simulator pipeline that renders moving driving scenes as sweeps under randomized sensor settings (beam count, horizontal resolution, rotation rate) and adds missing-return patterns that persist across frames. The second is a coarse-to-fine scene flow network, meaning a model that predicts a 3D motion vector for every point, that works on the range image, the sensor's native "unrolled" view. Each level of its pyramid is indexed by physical beam angle and carries a per-cell measurement time, so warping between scans uses each point's true time gap instead of one global frame interval. Because the network speaks in angles and seconds rather than pixel indices, one model trained in simulation should run on real sensors with different beam counts and rotation rates.

Core Claim

We make a falsifiable claim with three parts. First, a meaningful share of the synthetic-to-real error in LiDAR scene flow comes from three sensor properties that current synthetic training data lacks: per-point acquisition time within a sweep, the angular sampling grid of a particular sensor, and missing returns that persist over time. Second, a range-view hierarchical estimator that carries acquisition time through its pyramid and is trained on sweep-rendered, sensor-randomized simulation will beat point-based estimators trained on the same data. The advantage should show on real sweeps and especially on real sensors whose configuration was never seen in training. Third, the gains should concentrate where the physics says they should. For the timing component, that means near objects that cover a wide azimuth and move fast relative to the ego vehicle, plus the seam where a sweep begins and ends. For the dropout component, it means persistently non-reflective surfaces. If the gains are spread uniformly, or appear equally on data with no sweep timing, the claim is wrong.

How the Four Directions Fit

The spatiotemporal trade-off here is the LiDAR's own. At a fixed laser firing rate, points per sweep and sweeps per second trade against each other, and every sweep smears acquisition over its rotation period. Hardware-side answers to such trade-offs, like the hybrid high-speed/high-resolution camera rig of Li et al., fuse two physical sensors and depend on careful calibration. We take the opposite approach: we model the trade-off inside the simulator and randomize over it, so the network learns to handle any point on the resolution-rate curve.

Projection loss enters because we choose the range view for speed. GFNet shows that the range view hides distant surfaces behind near ones, while the bird's-eye view, a top-down grid, merges objects stacked at different heights. It fixes both with an "align-before-fuse" module that uses raw 3D coordinates to move features between views, raising the range-view branch on trucks from 32.4% to 69.9% mIoU. GFNet's alignment is a within-frame geometric correspondence with no time axis. We reuse it at the coarsest pyramid level, and extend it so each lifted cell also carries a timestamp. The range view then plays two roles: it is where occlusion hurts, and it is where time is naturally organized, since columns correspond to firing order.

Hierarchical motion estimation supplies the method. Coarse-to-fine warping handles large motions cheaply. PV-RAFT warns that errors at coarse levels propagate and that small fast objects vanish. FSF-Net shows that a crude top-down homography flow is already useful for 4D occupancy forecasting, although it assumes vertical motion away. Our pyramid is designed around PV-RAFT's criticism. Downsampling in the range view also merges points measured at different times, so each coarse cell keeps the time of its members rather than averaging it away.

The synthetic-to-real gap is the target. Two prior results constrain the design. Adversarial Masking shows that clean, complete synthetic scans give domain discriminators an easy shortcut, and that learning realistic point dropout lifts SynLiDAR-to-SemanticKITTI segmentation from 15.2% to 22.8% mIoU. That is still far from the 55.5% of a model trained on real labels, and it operates only on single frames. Saleh et al. translate synthetic top-down LiDAR images with CycleGAN, a pair of generators that learn to map between two unpaired image domains, to inject noise and motion distortion. This improves vehicle detection from 29.93% to 34.78% AP, but everything happens after flattening and involves no motion. Neither work treats the time structure of the sweep as something to simulate and model explicitly.

Research Program

The simulator side renders driving sequences in CARLA, the simulator behind the MDLS data of Saleh et al.. Actor poses give exact scene flow. Each sequence is rendered as a true sweep: every return is ray-cast at its own firing time using the scene state at that moment. For each training clip we draw beam count, azimuth resolution, and rotation rate from a range covering common spinning sensors, holding the firing rate fixed so resolution and rate trade off as they do in hardware. Missing returns come from a learned mask in the spirit of Adversarial Masking. In our version the mask is conditioned on surface identity and is correlated across consecutive sweeps, so a windshield that drops out in one sweep tends to drop out in the next. This is the temporally correlated dropout that the file's gap analysis notes no scene flow method has modeled. Ground-truth flow is defined at each point's own timestamp.

The estimator works on range images whose rows and columns are tied to physical elevation and azimuth angles, with each cell storing range, intensity, and acquisition time. Suppose point $i$ is measured at position $p_i$ at time $t_i = t_k + \phi_i/\omega$, where $t_k$ is the sweep start, $\phi_i$ its azimuth and $\omega$ the rotation rate. The network predicts a velocity $v_i$, and the displacement used to warp it toward a candidate match $j$ in the next sweep is

$$ d_{ij} = v_i \,(t_j - t_i), $$

instead of $v_i \Delta$ with a single frame interval $\Delta$. At each pyramid level, cells are pooled over fixed angular extents, which makes the level structure independent of the sensor's pixel count. Cells keep the time of their members, and the coarse-to-fine residual updates use the same time-aware warp. At the coarsest level, features are lifted to a top-down grid through the 3D-coordinate bridge of GFNet, so an object hidden in the range view can still be matched. The final output is per-point flow, and the budget is to fit within one sweep period (100 ms at 10 Hz). The main analysis is a stratified error study. We bin every evaluation point by angular extent of its object, speed relative to ego, range, distance from the seam, and dropout persistence, and check whether each gain appears in the bins that its mechanism predicts.

Baselines

The naive reference point is RMS-FlowNet trained only on FlyingThings3D and applied zero-shot. It reports an Out3D of 20.3% and Acc3DR of 93.8% on KITTI Scene Flow, processing 8,192 points in 77 ms. These are the numbers to beat under the standard protocol, and we will reproduce them before comparing.

The strongest existing competitors are RMS-FlowNet and PV-RAFT, run as point-based networks. PV-RAFT reports 0.0560 m EPE and a 21.63% outlier rate on KITTI Scene Flow 2015, and its official code is available as weiyithu/PV-RAFT. Point networks are the natural rival for two reasons. They are already indifferent to beam layout, and they have the best reported zero-shot transfer. For a fair comparison, both will be retrained on exactly our sweep-rendered, sensor-randomized, dropout-augmented simulation, with the same number of training iterations and the same hyperparameter search budget as our model. Our model beats them only if it improves Out3D and EPE on real data under this matched-data condition while staying within the sweep-period latency budget.

Four ablations isolate the key ingredient. The first gives our network timestamps only as an extra input channel, with the constant-interval warp. The second removes timestamps entirely and uses a pixel-indexed pyramid. The third swaps persistent dropout for independent per-frame dropout. The fourth removes the top-down lift. Each ablation must lose most in the strata its mechanism targets.

Why not just…?

Why not just deskew each scan with ego-motion before running an existing flow network? Correcting for the vehicle's own motion is standard practice and removes most distortion of the static world. That makes it a serious threat on aggregate metrics. It cannot correct moving objects, though, because their within-sweep motion is exactly what the network is trying to estimate. We will run ego-deskewed RMS-FlowNet on the same data. If it matches us on the moving-object strata as well as on static ones, the timing component of our claim fails.

Why not just train a point network on the new simulator data? This is the most dangerous alternative. Point networks do not care about beam layout, so sensor randomization might be all that matters and the range-view design might add nothing. The matched-data baseline above settles this directly. If RMS-FlowNet trained on our data equals our model, the contribution shrinks to the simulator pipeline, and we would report it that way. Our advantage should come from explicit time handling, and from speed relative to the more than 1.3-second point methods that the RMS-FlowNet paper benchmarks against.

Why not just resample every real sensor to one canonical range-image grid? Interpolating a 32-beam scan up to 64 rows invents returns and leaves timing untouched. It is still cheap, so we will include it as a cross-sensor baseline using our network without angle-indexed pooling.

Why not just adapt at test time with self-supervision? Per-scene optimization such as Neural Scene Flow Prior and FastNSF needs no synthetic data at all. We will run both on the same real evaluation clouds and report accuracy and runtime. These methods also assume instantaneous scans, so the timing strata test them too.

Proposed Experiments

The first experiment is a controlled simulation study that measures the timing gap in isolation. We train on instantaneous renders and test on sweep renders of the same scenes, then compare with training on sweeps. We also hold out entire sensor configurations, for example training at 64 beams and 10 Hz and testing at 32 beams and 20 Hz, to show that the synthetic sweep gap exists and that angle-indexed pooling transfers across configurations.

The second experiment is zero-shot transfer to KITTI Scene Flow, compared against the reported RMS-FlowNet and PV-RAFT figures and our matched-data retrains. The evaluation point clouds in this benchmark may not keep per-point acquisition order. If they do not, the timing component should give no gain there. We treat that as a built-in negative control, and attribute any KITTI improvement to the resolution and dropout components.

The third experiment tests transfer to a different real sensor: nuScenes, with a 32-beam LiDAR. We found no reported scene flow result on Lacuna for nuScenes, so evaluation there has two parts. One uses rigid flow derived from annotated object motion where available. The other is downstream: we replace FSF-Net's coarse homography flow with our predicted flow for Occ3D occupancy forecasting. That paper reports only relative gains, 9.56% higher mIoU than OccWorld for 3D forecasting of the first future frame, and we found no reported absolute value on Lacuna, so we will re-run FSF-Net ourselves as the reference.

The decisive check across all three experiments is the stratified error study. Our model should gain most on near, fast, wide-azimuth objects, on the sweep seam, and on persistently dropped surfaces. It should gain nothing on data without sweep timing. If the gains appear uniformly, or the matched-data point network closes them, we will have shown that the gap lies elsewhere. That would also be worth knowing, since current synthetic scene flow data assumes instantaneous capture without having tested whether that assumption costs anything.