Correlated Ego-Motion Error Randomization for Sim-to-Real LiDAR Segmentation

A spinning LiDAR takes about a tenth of a second to sweep the scene. Robots correct for their own motion during that sweep ("deskewing"), and they stitch successive sweeps together using an estimate of where the vehicle was. Both steps rely on the same estimated trajectory, so when that estimate is wrong the two errors are linked. A slightly wrong velocity shears each sweep in a particular direction, and the same wrong velocity misaligns consecutive sweeps by a matching amount. Networks trained in simulation never see this. Synthetic LiDAR is rendered as perfect snapshots with perfect poses. At deployment, real sweeps carry the leftover distortion of a real motion estimator.

We propose to randomize the error of the ego-motion estimate during simulation training, rather than randomizing the motion itself or adding independent per-point noise. Domain randomization means training on many randomly varied simulated conditions so the real world looks like one more variation. The error is drawn as a smooth random function of time, built from the motion prior that continuous-time odometry already uses. That single draw then produces the within-sweep shear and the between-sweep misalignment together. The claim to test is that this coupled, physically shaped error model transfers better than independent augmentations. The advantage should be largest for methods that pool information across frames, and those methods are exactly the ones prior work flags as fragile to pose error.

Core Claim and What Is New

The target is a problem stated openly in GOOD. GOOD does open-set test-time adaptation for LiDAR semantic segmentation from synthetic Synth4D to real SemanticKITTI. It clusters points into superpoints and adds a temporal pseudo-label branch that projects predictions across consecutive sweeps using the ego pose. Its authors list ego-pose dependency as a main limitation. GOOD motivates this proposal: it shows temporal consensus helps sim-to-real transfer, and it concedes that the consensus breaks when poses are wrong.

Two other papers show that the vulnerability is large and structured. FLAT builds a differentiable model of intra-sweep distortion by splitting a sweep into packets and applying an interpolated pose to each. It uses the model only to attack detectors. A 10 cm trajectory perturbation cut PointRCNN stage-2 AP by up to 75.3%. Dong et al. benchmark 27 corruptions, including motion-compensation errors. They report that generic augmentations such as Mixup and Dropout do not close the gap. That result constrains the design: whatever we randomize must reproduce the structure of motion error, not just its magnitude.

The method comes from continuous-time estimation. Burnett et al. represent trajectories with a sparse Gaussian process prior. A Gaussian process is a distribution over smooth functions; theirs uses a "white-noise-on-acceleration" prior. We reuse that prior generatively, as a distribution over estimation errors rather than as an estimator.

The obvious idea at this intersection would be "render synthetic LiDAR along randomized trajectories." We think that is weaker than it looks. Re-warping already-rendered points with per-point timestamps, which is FLAT's operation, is nearly equivalent over 100 ms. The reason to add randomization is to model the estimator's residual, coupled across the intra-sweep and inter-frame scales. No existing work does this. VICET and the spline SLAM of Droeschel and Behnke reduce distortion by optimization but train nothing. DexPoint and SIRA-PCR transfer point-cloud models zero-shot, but they treat clouds as static snapshots.

Research Program

Each synthetic sequence comes with a true trajectory $T(t)$. We sample a pose-error process $\delta\xi(t) \in \mathbb{R}^6$ (translation and rotation error, as a function of time $t$) from a Gaussian process whose acceleration is white noise with power spectral density $Q$. We then treat the corrupted estimate

$$\hat T(t) = \exp(\delta\xi(t)^\wedge)\, T(t)$$

as what the robot "believes." Here $\exp(\cdot^\wedge)$ maps the six-dimensional error to a rigid-body transform.

Every point $p_i$ has a timestamp $t_i$, taken from the renderer or assigned by azimuth. We deskew it with the believed pose instead of the true one, which produces

$$\tilde p_i = \hat T(t_{\text{ref}})^{-1}\, \hat T(t_i)\, T(t_i)^{-1} T(t_{\text{ref}})\, p_i,$$

where $t_{\text{ref}}$ is the sweep's reference time. The believed poses at successive sweep times are then used for multi-frame projection in the temporal branch. Labels come from the true geometry. This follows the asymmetric idea of Pinto et al.: privileged simulator information supervises learning, and the deployed model only sees what a real robot would see.

The design choice we want to test is calibration versus breadth. Broad randomization is known to make models overly conservative. So we fit $Q$ to real odometry residuals, comparing STEAM-LIO against the survey ground truth of the Newer College Dataset, and compare that against the same process with $Q$ inflated by 3× and 10×.

We will train a source segmentation network with this randomization on Synth4D and run GOOD's adaptation unchanged on top. That keeps the contribution isolated to the training distribution. As a second variant, we will feed the network the believed velocity. This tests whether conditioning on the believed motion helps beyond simple invariance.

Baselines

The naive reference point is a source-only model trained on clean Synth4D and applied directly to SemanticKITTI. The next reference is that same model with GOOD's adaptation. GOOD is the strongest existing method on this question: it is the only sim-to-real point-cloud method in the file that explicitly uses ego pose. On Synth4D→SemanticKITTI it reports 73.39% AUROC and 71.91% FPR95 for out-of-distribution detection, and a +4.83% mIoU gain over the source model (+1.93% over HGL). The file does not give GOOD's absolute mIoU, so we will re-run it and report our reproduction next to those figures.

Every variant uses the same backbone, the same Synth4D data, the same number of training steps, and the same hyperparameter search budget of equal trial count per method. Adaptation settings are identical, including GOOD's roughly 0.7 s per frame.

The key ablation replaces correlated errors with independent augmentations of matched magnitude. In one version, each sweep gets an i.i.d. pose jitter plus an i.i.d. linear shear, drawn at root-mean-square matched to the Gaussian process draws. In another, we apply Dong-style motion corruptions. A second ablation keeps the Gaussian process error only within sweeps and gives the temporal branch true poses, which tells us whether the coupling itself matters. We also include FLAT-style adversarial trajectory training as a worst-case competitor. An oracle upper bound uses clean, true-pose real data where it can be constructed.

Beating GOOD means improving mIoU and AUROC on clean SemanticKITTI by more than the seed-to-seed spread over three seeds. It also means degrading more slowly as injected pose error grows. The ablation criterion is stricter: the correlated model must beat the matched-magnitude independent model. If it does not, the core claim fails.

Why not just…?

Why not just deskew better upstream, with STEAM-LIO or VICET? Continuous-time odometry does shrink the error. STEAM-LIO reaches 0.304 m RMSE on the Newer College Dataset "Short" sequence at about 74 ms per frame. VICET cuts forward-translation error to 0.27 cm, but it needs a survey-grade prior map and a good initial guess. Neither drives the residual to zero, and FLAT shows that centimetre-scale residuals matter. This is still a real threat. We will run every method on sequences whose poses come from a strong odometer and on sequences with degraded odometry. If our gain disappears under strong odometry, we will state that the contribution is limited to cheap or aggressive-motion platforms.

Why not just add independent pose jitter and point noise? This is the most likely serious competitor. Dong et al. found generic augmentations insufficient, but they did not test matched-magnitude pose jitter. The ablation described above settles the question directly. We would publish a negative result if the ablation shows no difference.

Why not just drop the temporal branch so pose error cannot hurt? GOOD's gains partly come from temporal consensus, so dropping it gives up information. Still, at high pose error a single-frame model may win. We will plot single-frame and temporal GOOD against injected error level. The crossover point is itself useful, and our method should push it to higher error levels.

Why not train adversarially against FLAT-style perturbations? Worst-case training targets rare errors and risks exactly the conservatism that domain randomization warns about. It is included as a baseline, and we expect it to lose accuracy on clean data.

Proposed Experiments

The first experiment is the main result on Synth4D→SemanticKITTI, using mIoU on known classes, AUROC, and FPR95, compared against GOOD's reported figures and our reproduction.

The second is a controlled stress test. We inject Gaussian process pose error calibrated to the Newer College Dataset into SemanticKITTI at test time, at 0.5× to 5× the calibrated $Q$, and report error-versus-performance curves for all baselines. For comparison, we also inject FLAT-style smooth perturbations of 10 cm.

The third is a calibration study comparing calibrated $Q$ against 3× and 10× inflated $Q$. It checks whether matching the real residual beats simply randomizing broadly.

The fourth is a transfer check to 3D detection on KITTI-C and nuScenes-C motion corruptions. It asks whether the training recipe helps beyond segmentation. No reported result was found on Lacuna for detectors trained with motion-error randomization on these splits, so our numbers there will be new.

Two outcomes are decisive for the claim. The first is whether the correlated model beats matched independent jitter. The second is whether that margin grows when the temporal branch is active.