Time-Corrected Projection of Labels and Features in Moving Scenes

Many driving and robotics datasets avoid hand-labeling every video frame. Annotators label the scene once in 3D, as a point cloud, a mesh, or a set of boxes, and a camera model then pushes those labels into every image. The same operation, mapping a 3D point to a pixel, is how most camera–LiDAR fusion networks attach image features to LiDAR points. Both uses assume that the point and the pixel describe the same thing at the same moment. That holds for walls and roads. It fails for a car moving at highway speed. The LiDAR caught the car at one instant and the camera exposed its pixel at another, so the projected label or feature lands slightly behind or ahead of the object. The error is largest on exactly the moving road users we most need labeled.

This note proposes treating that misalignment as a measurable, correctable quantity instead of noise for a large network to absorb. Each 3D point carries a capture time and a velocity, taken from the static/dynamic split that dynamic reconstruction already provides. Before projection, the point is moved to the time its target pixel was exposed. A same-time depth check then removes points that are hidden at that moment. The claim is that this cheap correction recovers much of the accuracy currently attributed to heavy 4D reconstruction for label transfer, and to cross-attention for fusion, and that the recovered accuracy concentrates on fast-moving objects. If it does, both the annotation pipeline and the fusion design get simpler.

Core Claim

Label projection and feature fusion are one operation with one dynamic failure mode. For a 3D point $x$ captured at time $t_x$ with velocity $v$, we project it into the camera at the exposure time $\tau$ of the image row it falls on:

$$ u = \pi\big(K,\; T_{\text{cam}}(\tau)^{-1}\,[\,x + v\,(\tau - t_x)\,]\big) $$

Here $\pi$ is the pinhole projection with intrinsics $K$, and $T_{\text{cam}}(\tau)$ is the camera pose at time $\tau$, interpolated from ego poses. Because $\tau$ depends on the row the point lands on, the equation is solved by a few fixed-point steps. Static points have $v=0$ and reduce to standard projection. A second rule rejects a projected point when time-matched depth shows a closer surface along that pixel's ray. Time-matched depth means LiDAR returns from the same sweep, moved by their own velocities.

The falsifiable prediction is about where the error lives. Errors from naive projection should grow with the product of object speed and camera–LiDAR time offset. They should be near zero on static classes and on rigs with hardware-synchronized triggering. Our correction should remove most of that speed-dependent error.

This differs from prior work as follows. The closest systems either reconstruct full 4D scenes and render labels from them (DIAL-GS, which builds a 4D Gaussian scene with instance IDs from a static reconstruction plus a 2D tracker; EmerNeRF, which learns separate static and dynamic neural fields with emergent flow). Or they prune dynamic points out of the labeling problem (UniLiPs, which fuses LiDAR odometry with 2D foundation models and removes moving points from its background branch). Or they aggregate moving objects through annotated boxes but treat each sweep as a single instant (Occ3D). None of them isolates the time offset between sensors as an error source. None tests whether correcting it explicitly is enough.

Background and Positioning

Static 3D-to-2D transfer is well established and supplies our setting. KITTI-360-style label transfer annotates coarse 3D primitives in point clouds and projects them across hundreds of frames for about a seven-fold labeling speedup. ScanNet paints labels on meshes and renders them to every view. Both rely on a static world, and the research file names "dynamic violation" as the known breakdown.

Lifting 2D predictions into 3D constrains our design. Panoptic Lifting fits a neural field to flickering Mask2Former outputs and gains 10.6% scene-level panoptic quality on ScanNet. PCF-Lift represents point semantics as Gaussians to reconcile conflicting views, reaching 63.5% PQ-scene. Both state the static assumption explicitly. They show that 3D consistency helps, which motivates projecting from 3D rather than trusting per-frame 2D models. Neither tells us what happens when the 3D point itself is out of date.

The dynamic reconstruction literature supplies the velocities. Split4D separates static and dynamic Gaussians without trackers. RobustSplat shows that leaked transient pixels create permanent floating geometry in Gaussian splatting. That second result is a warning for any approach that bakes moving objects into a static model.

On the fusion side, PointFusion and MVX-Net concatenate image features sampled at projected pixels. SGFormer instead lets LiDAR queries attend over image features and reports 2–10% gains under occlusion and sensor-gap stress tests. Our hypothesis is that a meaningful part of that gain on moving objects comes from compensating for time misalignment, which a direct correction can supply at almost no cost.

Research Program

The work has three parts.

The first part is measurement. On nuScenes, every sensor sample has its own timestamp. Keyframes carry point-wise LiDAR semantic labels and 3D boxes with tracks. We quantify the projection error for each point as a function of class, object speed, and camera–LiDAR offset. The reference uses keyframe boxes interpolated to the exact camera exposure time. SemanticKITTI (SemanticKITTI) serves as a control with a tighter-synchronized rig. If naive projection error there is small and flat in speed, the mechanism is confirmed. If the error looks the same as on nuScenes, the time-offset story is wrong and the project changes direction.

The second part is a label transfer pipeline. Annotate only every $k$-th keyframe. Propagate those labels to the remaining LiDAR sweeps and all camera frames. Static points are aggregated across sweeps. Dynamic points are carried along tracked boxes, and articulated objects such as pedestrians get per-point scene flow. Every projection uses the time-corrected equation and the same-time visibility check. For the unlabeled setting, velocities come from estimated scene flow between consecutive sweeps instead of annotated boxes, and we report the gap between oracle and estimated velocity explicitly.

The third part is a fusion study. A PointFusion-style concatenation network is trained twice on nuScenes panoptic segmentation, once sampling image features at naive projections and once at time-corrected projections, with velocity estimated from the previous sweep. This is compared against SGFormer.

Baselines

The do-nothing reference is standard static projection. It uses the sweep's nominal timestamp, a single ego pose, and no visibility test. This is what most pipelines inherit from the static setting. Every method below must beat it on moving classes without losing on static ones.

The strongest box-based competitor is the Occ3D ground-truth pipeline (Occ3D). It separates static from dynamic points using annotated boxes, densifies them across frames, and applies visibility masking and image-boundary trimming. Moving objects are handled as rigid box motion at each sweep's timestamp. We reimplement it on the same nuScenes scenes with the same box inputs. Our method differs only in exposure-time advection and same-time visibility, so this comparison directly isolates our ingredient. There is no reported label-transfer accuracy for this baseline in the research file. Occ3D reports its downstream CTF-Occ model at 28.53 mIoU on Occ3D-nuScenes.

The strongest pseudo-labeling competitor is UniLiPs, run from its geometry-grounded dynamic decomposition on the same raw streams. No reported result found on Lacuna for UniLiPs label accuracy on nuScenes.

The heavy 4D competitor is DIAL-GS. Its scene is fit per sequence, labels are rendered into each camera, and the same keyframe labels are lifted onto its instance IDs. It reports 36.88 PSNR on reconstruction, but there is no reported label-transfer number. We give it its published per-scene optimization budget and report wall-clock time beside accuracy.

For fusion, the strongest existing method is SGFormer, which reports 80.9% PQ on the nuScenes validation set. Concatenation with time-corrected sampling counts as beating the comparison if it closes at least half of the gap between naive concatenation and SGFormer on moving-object PQ, at lower latency than SGFormer's reported roughly 100 ms. All fusion models use the same backbone, the same training schedule, and equal tuning budgets.

Ablations remove one component at a time: advection only, visibility only, and per-row exposure time replaced by the image's single mean timestamp. A final ablation swaps oracle velocities for estimated scene flow.

Why not just…?

Why not just ignore a band of pixels around moving objects? Eroding or ignoring boundary pixels of dynamic instances is the cheapest fix and a real threat. It throws away exactly the boundary supervision that small and distant objects need. It also cannot fix a label displaced by more than the band width. We include an ignore-band baseline tuned per speed bin. The decisive test is boundary quality and recall on fast and small objects. If the band matches our method there, we will report that the correction is unnecessary.

Why not just let cross-attention learn the correction? SGFormer shows attention is more robust than concatenation under stress. However, a learned network cannot correct the training labels themselves, and those are produced by projection. Learning the correction also costs model capacity and latency. The fusion study settles this directly. If time-corrected concatenation does not narrow the moving-object gap to SGFormer, the hypothesis that attention mostly fixes misalignment is false.

Why not just reconstruct the scene in 4D and render labels? DIAL-GS and EmerNeRF do this. EmerNeRF additionally removes positional-embedding artifacts from lifted foundation-model features. These methods require per-scene optimization and assume known poses. They depend on external trackers that fail under occlusion, as Split4D's comparison against tracker-dependent SA4D (0.893 versus 0.592 mIoU on Multi-Human) illustrates. They also risk floaters where motion masks leak (RobustSplat). If DIAL-GS-rendered labels beat ours by a clear margin, the extra cost is justified. We report that comparison as a cost–accuracy curve rather than a single number.

Why not just run SAM2 on every frame and skip projection? Per-frame 2D models flicker and switch identities. Motion4D shows that SAM2 alone (89.4 J&F on DyCheck-VOS) is improved by 3D-consistent prompting (91.7). Panoptic Lifting's double-digit gains come from removing 2D label noise. Per-frame masks also carry no 3D identity for the LiDAR-side labels.

Proposed Experiments

The first decisive check is the error-versus-speed curve. Naive projection error on nuScenes moving classes should rise roughly linearly with speed times sensor offset, because displacement equals velocity times elapsed time. It should be flat on static classes and on SemanticKITTI. Our correction should flatten the nuScenes curve.

The second check is label-transfer accuracy. Labels from every $k$-th keyframe, with $k$ ranging from 2 to 10, are propagated and scored against held-out keyframe labels. Scoring is done at the held-out keyframe's own LiDAR points, projected into its cameras, and reported as mIoU and panoptic quality split by static, slow, and fast classes. This is compared against static projection, the Occ3D-style pipeline, UniLiPs, and DIAL-GS.

The third check is downstream annotation efficiency. A LiDAR segmenter from Codes-for-PVKD (the evaluation backbone used by UniLiPs) is trained on each pipeline's propagated labels. We plot validation mIoU against the fraction of keyframes hand-labeled.

The fourth check is the fusion comparison against SGFormer's 80.9% PQ, broken down by speed bin.

A secondary transfer test regenerates Occ3D-nuScenes training targets with time-corrected image trimming and retrains CTF-Occ against its reported 28.53 mIoU. The prediction is gains on dynamic classes and none on static ones.

If none of the checks shows a speed-dependent effect, the conclusion is that sensor asynchrony is not the bottleneck. That negative result is also worth publishing, because it would redirect effort toward calibration drift and occlusion.