Tracking Sparse LiDAR Targets with the Beams That Miss

When a distant pedestrian or car returns only a handful of LiDAR points, today's 3D trackers try to infer its box from those few hits. Most of the evidence about where the object is and which way it faces is thrown away. Every laser beam that passes just beside the object and strikes the ground or a wall behind it says "the object is not here." Every beam that returns from in front of where the object should be says "something blocks the view." In the sparse regime, the beams that miss outnumber the beams that hit by an order of magnitude. On nuScenes, 69% of tracking frames have fewer than 15 points on the target (PointRePar), so this is the common case, not an edge case.

We propose to keep a small, explicit 3D model of each tracked object and to refine its pose by rendering that model into the LiDAR's own beam geometry, then comparing it with every beam in a window around the target, hits and misses alike. The model is a few hundred Gaussians, meaning soft 3D blobs of the kind used in Gaussian splatting (a fast rendering method that projects such blobs onto an image and blends them). The misses are only informative if the model has a back and sides, and those are almost never observed. A one-step shape-completion network therefore supplies the unseen geometry. Later free-space evidence carves away any completed geometry that it contradicts. The goal is a tracker that corrects heading drift on very sparse targets while running above 30 frames per second.

Core Claim

Under extreme sparsity, a target's pose is better constrained by free-space evidence from nearby non-returning beams than by its own points. Exploiting that evidence requires three things together: a completed shape model, so that predicted silhouettes and extents are meaningful; differentiable rendering in the sensor's range geometry, so the comparison is smooth rather than a hard nearest-neighbor lookup; and online carving, so that learned shape guesses are corrected by measurement rather than trusted. Each ingredient exists separately. No system combines them inside an online tracking loop, and we predict the combination specifically fixes heading error in the fewer-than-15-point regime, the failure mode that motion-centric and memory-based trackers leave open.

Relation to Prior Work

Three existing trackers already fight incompleteness, and they motivate the gap. M²-Track predicts inter-frame motion and then densifies the target by warping past points into the current frame. That cannot recover surfaces the sensor never swept, such as the back of a car driving straight away. PCET distils features from a teacher trained on dense merged clouds, completing shapes only implicitly in feature space. MVCTrack back-projects camera mask pixels using the nearest LiDAR depth. Its own analysis notes depth bleeding at object boundaries. All three use only returned points. None uses the absence of returns.

Two lines of work supply the method. EnerGS splits space into occupied, free and unknown regions to stop Gaussian splatting from placing spurious blobs under incomplete LiDAR. It explicitly assumes a static scene, which moving objects violate. We apply the same three-way partition in each target's own moving coordinate frame, so the static assumption holds by construction. 3D MeanFlow completes a partial shape in a single network evaluation of about 5 ms. Its PointPlug variant gates completion to the most useful detection proposals, but it was only evaluated on single-frame detection, never in tracking. We reuse it as the source of unseen geometry. Its output becomes a hypothesis to be tested against free space, not a final answer.

Two works constrain the design. SplAttN shows that hard projection cuts gradient flow: a hard-projection competitor changed by +0.4% when images were removed, while its soft Gaussian splatting model dropped 26.1%. This argues for soft rendering over nearest-neighbor association. At 40.75 ms per object, it also shows that a heavyweight cross-modal completion network cannot run per frame. GSGTrack is the closest render-and-compare tracker. It tracks objects as Gaussians via differentiable silhouettes in monocular RGB, reaching 64.60% ADD-S on HO3D. It cannot complete unseen backs and does not use LiDAR. We differ in rendering into beams rather than pixels and in using a completion prior plus free-space carving. The idea of predicting where a ray terminates, as in TermiNeRF, is what we compute per beam, here in closed form from the Gaussians rather than with a learned network.

Research Program

Object model. Each target is represented by roughly 256 Gaussians in its own coordinate frame. Each Gaussian carries a provenance confidence $c_i \in [0,1]$. Gaussians fit to accumulated observed points start near 1. Gaussians proposed by the completion network start at a lower value tied to the network's disagreement across sampled completions.

Beam window. At each frame we take all beams whose directions fall within the angular footprint of the predicted box, enlarged by a margin. That is a few hundred beams for a 32-beam sensor at long range. Beams are reconstructed from ring index and azimuth; Waymo provides range images natively. Each beam is classified as a hit if its return lies inside the enlarged box, or as a pass-through if it returns from behind the box or not at all.

Pose refinement. Given an initial pose $T$ from a fast motion head, we minimize

$$E(T) = \sum_{b \in H} \rho\big(\hat d_b(T) - d_b\big) + \lambda \sum_{b \in P} \alpha_b(T)^2 .$$

The symbols are defined as follows:

  • $H$ is the set of hit beams and $P$ the set of pass-through beams.
  • $d_b$ is the observed range of beam $b$.
  • $\hat d_b(T)$ is the expected termination depth that the confidence-weighted Gaussians render along beam $b$ when the object sits at pose $T$.
  • $\rho$ is a robust Welsch penalty, borrowed from EnerGS's occupied-space term.
  • $\alpha_b(T)$ is the opacity the model accumulates along beam $b$ in front of that beam's observed return.
  • $\lambda$ balances the two terms.

The first term pulls the surface onto the few hits. The second penalizes any pose at which the model would have blocked a beam that in fact went through. Rotations that swing the completed body into observed free space become expensive, and this is the lever against heading drift. We run a few Gauss–Newton iterations in yaw, translation and optionally pitch. The per-beam rendering is a one-dimensional version of tile-based splatting and should cost on the order of a millisecond. That is a target to verify, not a result.

Shape maintenance. When a completed Gaussian receives repeated pass-through evidence after pose refinement, its confidence decays and it is eventually removed. New hits spawn observed Gaussians. The completion network is re-queried only when observed coverage changes substantially, reusing PointPlug's gating idea. This keeps its roughly 5 ms cost off most frames. The result is a model that reconciles memory and hallucination. History fills what was seen, the prior fills what was not, and free space arbitrates between them.

Initialization. The coarse pose comes from an existing tracker head. That makes every experiment a clean "with versus without render-and-compare refinement" comparison. We use M²-Track at 57 FPS and BEVTrack at 200 FPS.

Baselines

Unrefined initializer. The naive reference point is the initializing tracker itself, M²-Track or BEVTrack, run unchanged on the same splits. Our refinement is a drop-in stage, so any gain over the tracker that seeds it is attributable to the stage. Beating it means higher Success and lower heading error in the fewer-than-15-point bin at no worse than a 10 ms added cost per target.

MBPTrack. The strongest LiDAR-only competitor is MBPTrack. It propagates target masks with a memory network and uses the first-frame box as a geometric prior. It reports 70.3% mean Success on KITTI at 50 FPS. It is the strongest because it already exploits both temporal memory and a shape-like box prior, the two things our method most resembles. We will retrain it with its released recipe on identical splits, time both methods on the same GPU, and require our full system to exceed 70.3% mean Success on KITTI. The larger margin should appear in the sparse bins on nuScenes.

MVCTrack. On nuScenes the strongest reported number is MVCTrack's 61.20 mean Success and 73.22 mean Precision at 32.1 FPS. It uses camera images, which our primary system does not. Matching it with LiDAR alone would be the claim. A camera-silhouette term added to $E(T)$ is a secondary experiment.

PCET. For implicit completion we compare against PCET, which reports 64.8% Success on KITTI at 32.8 FPS.

Reported numbers we cannot rely on. No reported result was found on Lacuna for BEVTrack's or PointRePar's per-category Success and Precision, or for any tracker's heading error by point-count bin. We will compute these ourselves.

Ablations. Four ablations isolate the key ingredients:

  • removing the pass-through term ($\lambda = 0$);
  • replacing the completed shape with observed-only Gaussians;
  • replacing it with a cuboid of the known box size;
  • replacing soft beam rendering with hard nearest-point association.

All variants share the initializer, the iteration count and a hyperparameter search of equal size.

Why not just…?

Why not just accumulate past points, as M²-Track does? Accumulation cannot place geometry on surfaces the sensor never saw. Without that geometry, the pass-through term has nothing to push against on the unseen side. We test this directly with the observed-only-shape ablation. We will report it separately for targets that do and do not rotate relative to the sensor during the sequence, since accumulation should suffice only for the rotating ones.

Why not just plug 3D MeanFlow completion into a tracker and regress the box from the completed cloud? This is the obvious PointPlug-for-tracking system and a serious threat. We will build it: the same completion network, with the completed cloud fed to M²-Track's box-refinement stage. Our claim is that completion without free-space testing turns template hallucinations into confident wrong headings. If this baseline matches ours in the sparse bin, the free-space mechanism adds nothing and we will say so.

Why not just give a learned tracker a visibility channel, marking free versus occupied voxels in its bird's-eye-view input? This is cheap and could capture much of the benefit. We will train BEVTrack with an added ray-traced free-space channel, under the same budget. Two concerns motivate explicit refinement. Bird's-eye-view compression discards the vertical structure that separates grazing beams. A network must also learn a geometric test that our objective states exactly. Which matters more is an empirical question that this comparison settles.

Why not just use the known box as the shape? A cuboid already has a back and sides, and MBPTrack shows box priors help. The cuboid ablation checks whether completion is needed at all. We expect cuboids to over-carve for pedestrians and to misjudge extent for irregular vehicles. If cuboids match completed shapes on cars, we will narrow the completion claim to non-box-like categories.

Proposed Experiments

We evaluate on KITTI, nuScenes and Waymo single-object tracking with standard Success and Precision. Every metric is also broken down by target point count (0–15, 15–50, above 50) and by range (above 50 m). We add mean absolute heading error, because the hypothesized mechanism acts on orientation. Latency is reported per target and for ten simultaneous targets, on one GPU, with the completion calls counted.

The decisive check is the pass-through ablation in the fewer-than-15-point bin on nuScenes. If removing the miss term does not raise heading error there, the core claim is wrong. The second check is the three-way shape comparison of observed-only, cuboid and completed-with-carving. It tells us whether the learned prior earns its cost. The third is a cross-dataset test: train the completion prior and tune $\lambda$ on KITTI, then run on Waymo without retuning. This tests whether an explicit sensor-geometry objective transfers across LiDAR types better than learned trackers, which the file identifies as a weakness of density-tuned models. We will also report the fraction of completed Gaussians carved away over each track, to measure how often the prior was wrong and was corrected.