A LiDAR detector trained in a simulator sees clean geometry. On a real car it sees rain returns, beams that come back empty from dark or wet surfaces, jitter, and density that falls off with distance. Almost all work on this synthetic-to-real gap treats each scan as a single snapshot. A real sensor, however, produces a stream, and in that stream the two kinds of content behave differently over time. Once you account for the car's own motion and the motion of other objects, the shape of a parked van is nearly the same from one sweep to the next. A raindrop return, a jitter pattern or a random dropout is not. Our proposal is to use that difference in time as the bridge between simulation and reality. A slow branch of the network holds object semantics and is trained so that its features stay predictable across sweeps. A small fast branch holds motion. The predictability requirement needs no labels, so it can be imposed directly on unlabeled real driving logs. That gives an adaptation signal that current sim-to-real detectors, which work one frame at a time, do not use.
Put in two sentences: sensor noise flickers from sweep to sweep while objects persist, so a network whose semantic features are forced to be temporally predictable should learn to ignore noise. Because predictability can be measured without labels, the same constraint also adapts a simulator-trained detector to real unlabeled sequences.
Core Claim
We claim three things. First, splitting a multi-frame LiDAR detector into two asymmetric branches, and requiring the slow branch to be predictable across time after motion compensation, closes more of the simulation-to-real gap than single-frame uncertainty-aware self-training. Second, the same split degrades less than multi-frame baselines of equal cost under per-sweep corruptions that vary from frame to frame. Third, the gain comes from the temporal nature of the constraint. The same consistency loss applied between two augmented copies of a single frame should not match it.
The claim also predicts where it fails. Artifacts that persist over time should not be helped: distance-dependent density falloff, occlusion holes that move with the object, and above all a frozen sensor buffer, which is perfectly "persistent" by construction. These predicted failures are part of the test, not something to hide.
Where This Sits
The architecture comes from video recognition. SlowFast Networks for Video Recognition processes a few frames at full capacity in one pathway and many frames at roughly one-eighth the channels in another. The narrow fast pathway is forced to spend its capacity on motion. That design reached 79.8% top-1 on Kinetics-400 with the fast pathway using about 20% of compute. We reuse this asymmetry, but the goal is different. SlowFast splits by rate to save compute and model motion on clean RGB. We split by rate so that frame-transient sensor artifacts have nowhere to live in the semantic branch.
Existing 4D point models motivate the change by showing what symmetric processing does. Point 4D Transformer Networks builds metric neighborhoods in $(x,y,z,t)$. It is sensitive to the size of the temporal window, and it was never tested under corruption. 4DPC²hat pairs a per-frame Point-BERT encoder with a bidirectional Mamba sequence model, a model that processes long sequences in linear time. It is trained on Objaverse animations whose points are tracked one-to-one across frames, a property no physical sensor has.
Two driving papers constrain the design. VADet shows that stacking more sweeps densifies static objects but smears moving ones. It picks the number of sweeps per object from a hand-built lookup table keyed on velocity and density, with no notion of noise. Our fast branch plays the role of its constant-velocity motion model, but it is learned. Our persistence constraint replaces its lookup table with a learned, noise-sensitive weighting. Benchmarking the Robustness of LiDAR-Camera Fusion documents stuck and lagged frames and finds that performance falls linearly with freeze rate. This is the case where persistence is actively misleading, so we must test it.
On the sim-to-real side, CTS is the closest method. It uses a student-teacher loop with corner-format aleatoric uncertainty, a per-box estimate of label noise, and fixed anchors. It lifts CARLA3D→KITTI moderate $AP_{3D}$ from 4.06% to 45.28%, but it uses only single sweeps. Its adaptation signal comes from pseudo-labels, whereas ours comes from temporal predictability, and the two can be combined.
Two papers set the bar for what augmentation and filtering achieve. Common Corruption Robustness of Point Cloud Detectors finds an average 11.01% Car AP drop across 8 detectors on KITTI under 25 corruptions. In that study, physics-aware augmentation helps only slightly, and nearest-neighbor outlier removal can hurt. AdaptPoint shows that corruptions adapted to each sample beat one-size-fits-all augmentation on static shapes.
Research Program
The detector works on a bird's-eye-view grid, a top-down 2D feature map of the scene. The slow branch encodes a sparse set of sweeps, for example every fourth sweep of a 16-sweep window, each separately at full width, into grids $s_t$. The fast branch encodes all 16 sweeps at one-eighth width and predicts a per-cell displacement field, the "flow". Lateral connections pass fast features into the slow branch, as in SlowFast.
Motion compensation uses two sources: the known ego pose, and the fast branch's predicted flow for moving cells. This defines a warp $W_{t\leftarrow t-k}$ that carries slow features from sweep $t-k$ to sweep $t$. Training minimizes
$$ \mathcal{L} = \mathcal{L}_{\text{det}}(\text{sim}) + \lambda \sum_{t} \big\| h(s_t) - \mathrm{sg}\big[W_{t\leftarrow t-k}\, h(s_{t-k})\big] \big\|^2 + \mu\, \mathcal{L}_{\text{var}}. $$
Here $\mathcal{L}_{\text{det}}$ is the usual detection loss on labeled simulator sequences. $h$ is a small projection head. $\mathrm{sg}$ stops gradients into the target. $\lambda$ weights the persistence term. $\mathcal{L}_{\text{var}}$, weighted by $\mu$, keeps per-channel variance of $h(s_t)$ above a floor, so the trivial solution of constant features is ruled out.
The persistence term is applied to both simulator sequences and unlabeled real sequences. On simulator sequences we inject corruptions drawn independently per sweep, reusing the rain, snow, noise and density families of the KITTI corruption benchmark. On real sequences the sensor supplies the noise.
Three design choices need study. The first is how much fast-branch capacity is enough to explain motion without also absorbing noise, which we sweep across channel ratios. The second is whether gradients from the persistence loss should flow into the fast branch's flow, since the fast branch could cheat by "explaining away" noise as motion. We compare detaching the flow against a flow smoothness prior. The third is the temporal stride $k$. Its failure mode mirrors the temporal-radius sensitivity reported for Point 4D Transformer Networks.
We also build a temporal corruption suite. The KITTI and Waymo corruption families are applied independently per sweep (transient noise) or held fixed across the window (persistent noise). To this we add stuck-frame and lag corruptions following nuScenes-R. The persistent versus transient split is what makes the claim falsifiable.
Baselines
The naive reference is the same multi-frame backbone trained only on labeled simulator sweeps, with the sweeps simply stacked and no adaptation. This measures the raw gap.
The strongest sim-to-real competitor is CTS. The reported figure is 45.28% moderate $AP_{3D}$ on CARLA3D→KITTI and 45.87% on CARLA3D→Lyft (CTS). Our method needs sequential simulator sweeps, and the file does not say whether the released CARLA3D contains them. If it does not, we render sequences with matching simulator settings. We then rerun CTS single-frame and a multi-frame CTS variant on our split, with the same backbone, the same unlabeled target logs, and the same number of tuning trials. The 45.28% figure serves only as a sanity check that our CTS reproduction is faithful. Beating CTS means higher moderate $AP_{3D}$ than multi-frame CTS on both targets, with the gap exceeding seed-to-seed spread over three seeds. Combining CTS pseudo-labels with our persistence loss is also reported, because additive gains would be the most useful practical outcome.
The strongest multi-frame competitor is VADet. It reports 76.6 L2 AP on Waymo validation and 79.8 on test with a VoxelNeXt backbone (VADet). We run it with the same backbone and random aggregation training. We also give it a fair upgrade: its lookup table is refit on corrupted training data, so it can adapt aggregation to noise. On clean Waymo we aim to stay within about one L2 AP point of VADet. Beating it means a smaller relative drop under transient temporal corruptions. For temporal corruptions we found no reported result on Lacuna, so all corrupted numbers come from our own runs.
The ablation that isolates the key ingredient keeps the architecture, the augmentations and the loss fixed. It changes only the persistence pairs: two independently corrupted copies of the same sweep, instead of motion-compensated sweeps at different times. A second ablation removes the asymmetry and applies the persistence loss to a symmetric backbone of matched compute.
Why not just…?
Why not just train an existing multi-frame detector with heavy simulated corruptions? This is the most serious threat. The published evidence is mixed rather than decisive. Physics-aware augmentation improved robustness only slightly on KITTI corruptions (Common Corruption Robustness). Robust finetuning did not remove LiDAR dependency in fusion models (Benchmarking the Robustness of LiDAR-Camera Fusion). Both, however, are single-frame results. Our augmentation-matched ablation settles the question directly, since our method uses the same corruptions. If the same-frame consistency variant matches the temporal one, the temporal claim fails and we would report the contribution as an augmentation recipe. Augmentation also cannot use unlabeled real logs, which our loss can.
Why not just average per-frame predictions over time? If noise really is temporally incoherent, averaging the outputs of a single-frame detector over a few sweeps, after tracking, exploits the same fact at almost no cost. Here is where we expect it to fall short. Output averaging cannot recover geometry that no single sweep sees completely, such as a car half-hidden by dropouts in alternating sweeps. It also cannot supply a training signal on real unlabeled data. We run a tracked late-fusion baseline, which smooths box scores and positions across sweeps with the same window. If it matches us on detection accuracy under transient noise, the architectural argument is weak and we would say so.
Why not just filter outliers before a standard temporal model? Nearest-neighbor outlier removal showed limited benefit and sometimes hurt, because it deletes sparse valid returns from distant objects (Common Corruption Robustness). We include it as a preprocessing baseline mainly to confirm this finding holds in the multi-frame setting. We expect missed detections at long range to rise.
Why not just use VADet with a noise-aware lookup table? The noise-aware VADet is included precisely to test this. Choosing the number of sweeps per object is a coarse control. It cannot keep persistent points while dropping transient ones inside the same window. If refitting the table closes the gap, then learned persistence adds little beyond aggregation choice.
Decisive Experiments
The main experiment is simulator-to-real transfer, CARLA-style sequences to KITTI and Lyft, scored by moderate $AP_{3D}$ against the baselines above. Clean-domain accuracy on Waymo is reported as a guard against sacrificing ordinary performance.
The mechanistic test uses the temporal corruption suite on KITTI and Waymo. The prediction is sharp: our advantage over augmentation-matched and late-fusion baselines should be large under transient corruptions and roughly zero under the same corruptions held fixed across the window. A uniform advantage would suggest the gain comes from something other than persistence.
We also report error type, following the benchmark's split into false, misclassified and missed detections. Rain clutter mostly inflates false detections, so that is where transient-noise suppression should show up.
Finally, the stuck-frame corruption from nuScenes-R is run at increasing freeze rates. We expect our slow branch to be fooled. We then test whether one simple fix restores the linear-degradation baseline or better: flag disagreement between the fast branch's predicted flow and the known ego motion.
As a small controlled study, we inject per-sweep corruptions into the real depth videos of NTU RGB+D 120, where P4Transformer reports 93.5% on clean data. This checks whether the persistence effect carries over beyond driving detection to recognition from point cloud video.
References
- SlowFast Networks for Video Recognition
- Point 4D Transformer Networks for Spatio-Temporal Modeling in Point Cloud Videos
- 4DPC$^2$hat: Towards Dynamic Point Cloud Understanding with Failure-Aware Bootstrapping
- VADet: Multi-Frame LiDAR 3D Object Detection Using Variable Aggregation
- Benchmarking the Robustness of LiDAR-Camera Fusion for 3D Object Detection
- CTS: Sim-to-Real Unsupervised Domain Adaptation on 3D Detection
- Common Corruption Robustness of Point Cloud Detectors: Benchmark and Enhancement
- Sample-adaptive Augmentation for Point Cloud Recognition Against Real-world Corruptions