Learning to Complete Distant Moving Objects from Their Own Future

A self-driving car's laser scanner (LiDAR) sees a car 70 meters away as a handful of points, often fewer than ten. Recovering that car's full 3D shape, or even its orientation and size, from so little evidence is very hard. Two obvious remedies each fail on their own. Stacking scans over time works for buildings but smears anything that moves. Learned shape-completion networks, which guess the unseen parts of an object, are trained mostly on clean synthetic models and break down below a few dozen points. Camera images are dense at every range, but every method that uses images for shape completion assumes a single static object in a tidy, centered pose.

This note proposes to exploit a property of driving logs that existing work ignores. Many objects first seen far away are later seen up close, because the ego car approaches them or they approach the ego car. Their later, dense, multi-angle near-range scans give free real-world ground truth for their earlier sparse appearances. We would train a model that runs only on past and present frames. It aligns a distant moving object's few LiDAR points across time using camera tracking, then completes its shape with image-guided learned priors. Its targets come from the same physical instance observed later at close range. The claim to test is that this pairing of cross-range self-supervision with camera-anchored motion alignment turns far-field completion from a synthetic-benchmark exercise into a measurable real-world capability.

Core Claim

Far-field completion of moving objects fails for two separable reasons, and each has a specific fix. The first is motion. With fewer than ten returns per sweep, estimating an object's rigid motion from the points alone is ill-conditioned. Clustering-and-registration pipelines such as Dynamic 3D Scene Analysis by Point Cloud Accumulation rely on enough density to segment instances and regress per-object transforms. We replace point-only registration with a ground-plane-constrained motion estimate (yaw plus planar translation, three degrees of freedom for vehicles). It is solved from dense image correspondences inside the object's image region, with metric scale anchored by the few LiDAR depths. Three unknowns per frame, constrained by hundreds of pixel matches, is a well-posed problem even when the 3D point count is tiny.

The second reason is supervision. Synthetic training sets do not contain far-field scanline patterns. Accumulation-based pseudo ground truth, as in TraPCC, contains only surfaces visible from the original vantage. Cross-range pairing supplies targets that are both real and seen from different angles, since the near-range pass typically views sides the far-range pass never did. What is new is the combination. No existing method fuses camera features with temporally aligned sparse LiDAR for shape completion of moving objects, and none trains far-field completion against the same instance's own future near-field geometry.

Research Program

The first component is a dataset construction pipeline. It runs on the tracked training logs of Waymo, nuScenes, and Argoverse 2, all of which provide persistent track identities. Argoverse 2 matters most because its annotations extend to 150 meters. For each vehicle track, we split observations into a far segment and a near segment, and accumulate the near segment in the object frame using annotated boxes to form a dense target. We store a per-target visibility mask: the set of surface regions that any near-range sweep actually observed. Losses and metrics count only these regions, in the spirit of TraPCC's Modified Chamfer Distance, so the model is never penalized for parts nobody saw. Annotated boxes are used only to build targets offline. At test time the model sees tracker outputs, not ground truth.

The second component is the motion module. Given the object's image crops over a short causal window, it matches pixels across frames with a frozen 2D correspondence model. It lifts matches that land near projected LiDAR returns to metric depth, and solves for the three-degree-of-freedom rigid motion with a robust least-squares fit. It then transforms all past points into the current object frame. This adapts the rigid-instance view of the Huang et al. accumulation pipeline, but replaces its density-dependent clustering and transform regression with camera evidence. It also generalizes the pose-disturbance alignment of Temporal Point Cloud Completion with Pose Disturbance, which assumes a cropped instance and refines pose from geometry only.

The third component is the completion network. It encodes the aligned points, tagging each with its timestamp and range. Through cross-attention, as in XMFnet, it attends to image features from a frozen DINOv2 encoder, following Leveraging Pretrained Representations for Cross-Modal Point Cloud Completion. Image features are sampled only at pixels that pass a depth-consistency check against the LiDAR points. This addresses the foreground-background smearing that Sparse Fuse Dense had to handle with a dedicated convolution. The decoder borrows the few-point design lessons of FSC. We initially restrict scope to vehicles, because the rigid, ground-plane motion model holds for them. Pedestrians and cyclists are a later extension, not a claim.

Baselines

The naive reference is the raw current sweep inside the tracked box, plus plain ego-motion-compensated accumulation of past sweeps. This shows how much moving-object smear costs and sets the floor.

The strongest competitor is box-aligned accumulation followed by FSC. Past sweeps are transformed into the object frame using the online tracker's boxes, then completed by FSC retrained on our cross-range pairs. This pipeline uses the same targets and the same tracker as our method, but no camera. FSC is the right opponent because it is the only method in the file built for the few-point regime and evaluated on real scans: it reports a minimum matching distance of 0.239 on KITTI versus 0.392 for AdaPoinTr. We will also run it trained on ShapeNet only, to measure the value of the data alone.

Two further baselines cover the other design choices. The first is TraPCC, retrained with its own GT-Pool; it reports a mean Chamfer Distance of 0.0973 on KITTI. The second is per-frame cross-modal completion, the Kale et al. variant of XMFnet applied to single-frame crops plus image patches, which isolates "camera without time."

Every baseline gets the same logs, the same tracker, the same visibility-masked metrics, and an equal hyperparameter-search budget on a shared validation split. No reported result was found on Lacuna for far-field completion of moving vehicles on Argoverse 2 or Waymo, so all numbers there will be produced by us under one protocol.

Two ablations isolate the key ingredients. Swapping the camera-based motion module for point-only registration, while keeping image features in completion, measures the camera's role in alignment. Removing image features from completion, while keeping camera motion, measures its role in shape. Beating the strongest baseline means a lower visibility-masked Chamfer distance and higher F-score in the 50–80 m and 80–150 m bins, with the gap growing with range and holding with tracker boxes rather than annotated ones.

Why not just…?

Why not just accumulate along tracker boxes and run a few-point completer? This is the real threat, and it is our strongest baseline. If tracker boxes were accurate at range, camera-based motion would add nothing. The pose-disturbance study shows completion degrading sharply under alignment noise: PCN reaches a Chamfer Distance of 14.28 under disturbance versus 6.64 for explicit temporal alignment (Shi et al.). Far-field boxes are exactly where such noise is largest. The motion ablation settles the question. If box-aligned accumulation plus FSC matches us with tracker boxes, we will report that the camera's value lies only in completion features, or nowhere.

Why not just run static image-guided completion on each frame's crop? ViPC, XMFnet, and CDPNet were developed only on ShapeNet-ViPC, using centered canonical inputs. Their reported gains, a Chamfer Distance of 1.443 for XMFnet versus 3.308 for ViPC, say nothing about inputs of five points with jittered pose. The per-frame Kale et al. baseline measures this directly. We expect it to collapse in the far bins, because one frame gives too few points to orient the shape.

Why not just densify with depth completion or monocular depth? Sparse Fuse Dense documents smearing at object boundaries in pseudo-point clouds. Far3D shows depth error growing nonlinearly past 50 meters, which is why it needs range-modulated denoising. We will nonetheless include SFD-style pseudo-points inside the box as an input variant. If they help, they become an input, not a competitor.

Why not just skip completion, since detection barely benefits? TraPCC's completion filter moved SECOND's KITTI Hard AP only from 65.91% to 65.98%. That result constrains our downstream claim. We therefore do not promise detection gains. We test whether completed shapes reduce heading and size error for distant tracks, a narrower quantity that box regressors estimate poorly from a few points.

Proposed Experiments

The main experiment trains on the Argoverse 2 and Waymo training logs and evaluates on held-out logs. It reports visibility-masked Chamfer distance and F-score per range bin, for objects whose later near pass exists in the test log and is hidden from the model. The first decisive check is the range curve. Our advantage over box-aligned FSC should be near zero below 30 meters and widen beyond 50. A flat gap would suggest we are only learning a better generic prior.

The second check is the vantage test. On tracks where the near pass reveals a side the far pass never saw, we score only those newly revealed regions. Accumulation-only methods cannot place geometry there, so this check separates learned completion from stacking.

The third check is cross-dataset transfer: train on Argoverse 2 and test on nuScenes's sparser 32-beam LiDAR. This probes whether cross-range supervision avoids the synthetic-to-real failure that FSC and TraPCC inherit from their training data.

Finally, as a modest downstream test, we feed completed shapes to a box-refinement head on top of Far3D, which reports 24.4% mAP on Argoverse 2, and measure heading and size error beyond 80 meters. A null result here would be reported as such.