Objects Are What the Scene Cannot Predict: Context-Inpainting Residuals as a Motion-Free Teacher for LiDAR Detection

Training a 3D object detector for self-driving requires people to draw oriented boxes around objects in LiDAR point clouds, the sparse 3D scans produced by a spinning laser. Each box costs about 110 seconds of human time, against 5 seconds for a single click (ViT-WSS3D). The best label-free alternatives find objects because they move. That works for traffic, but it is blind by construction to things that never move: traffic cones, barriers, signs, lost cargo. We propose a different source of free supervision, borrowed from reconstruction-based anomaly detection, the family of methods that learns what "normal" data looks like and flags whatever the model fails to reproduce. We train a model to fill in masked regions of a LiDAR scan from the surrounding scene. Ground, walls and vegetation can be predicted from their context. A compact object usually cannot, because nothing around an empty patch of road tells you a cone stands there. The gap between the observed points and the filled-in background is therefore a per-point objectness signal that does not depend on motion or class.

The second ingredient comes from sensor physics. Once the model has predicted the background behind an object, the region between the object's first laser returns and the predicted background surface bounds how deep the object can be. That gives a physically grounded limit on box length for the roughly 65% of objects that are only partly visible (CPD). We use the residual to score and fit 7-degree-of-freedom pseudo-boxes (center, size and heading angle) and to weight a student detector's training loss. We test whether this closes the gap on never-moving classes where motion-based methods have nothing to offer.

Core Claim

We make three connected claims. First, a background inpainter trained with masked-context reconstruction yields point residuals that separate compact objects from scene structure better than ground removal plus clustering, especially for static objects. Second, the predicted-but-unobserved background behind a cluster gives a per-object depth bound that reduces box-size error on truncated objects. Third, pseudo-labels built from these two signals train detectors that beat motion-based and heuristic-prototype teachers on classes that never move, while staying competitive on vehicles.

We claim nothing about moving traffic beyond parity. The honest expectation is that most of the gain concentrates on static and never-moving classes.

Relation to Prior Work

Two papers already put reconstruction and LiDAR together, and they supply our method.

COVAL inserts synthetic objects into real scans and pairs a "factual" occluded view with a "counterfactual" view that keeps the hidden background. A sparse network then learns to reconstruct occluded points. It reaches 89.00% point AP and a panoptic quality of 57.95 on the Spotting the Unexpected (STU) benchmark. We reuse its paired-view construction as training signal for our inpainter. COVAL, however, stops at point masks. It never fits boxes or trains a detector, and it degrades on low-profile objects and at far range.

MOVES translates dynamic scans into static backgrounds with a GAN in order to segment movable objects for mapping. It also stops at point masks. It trains the static target from observed dynamics, whereas we define background purely by predictability from context. MOVES reports little gain on sparse 16-beam data, and we take that as a warning about far-range regions.

The detection side supplies the gap and the competitors.

LISO mines moving clusters with self-supervised scene flow and trains a single-frame detector on them. Its student transfers surprisingly well to still objects: 0.322 AP on still objects versus 0.350 on moving ones on Waymo. This result constrains our proposal. It means the "static blindspot" argued in Tangemann et al. and Croitoru et al. mostly bites on classes that are never seen moving, not on parked cars. We therefore make never-moving classes the main test.

CPD clusters aggregated multi-frame points, which do not need motion, and scores them with hand-set commonsense rules. It then regularizes incomplete boxes toward prototype sizes, reaching 32.13% AP on Waymo Vehicle L2. It is our strongest competitor. Its known weakness is rare classes, where complete exemplars are scarce.

UA3D weights student losses by coordinate-wise uncertainty, estimated from disagreement between two detector heads. We replace that learned disagreement with a residual that is tied to sensor geometry, and we compare the two directly.

The 2D anomaly work supplies design lessons rather than components. DRÆM showed that a learned discriminator over the concatenated input and reconstruction beats raw subtraction. TransFusion showed that iterative mask feedback resists the "identical shortcut", where a model reproduces the anomaly and the residual vanishes. Its 0.34 s per-sample cost is irrelevant offline, since labeling runs once over a stored dataset.

Research Program

The inpainter. We build it on a sparse-convolution backbone, the MinkNet backbone COVAL uses. Given a scan, we mask a vertical column or a view-frustum wedge and ask the network to predict occupancy and point positions inside it, using only points outside the mask.

It receives two kinds of supervision. Self-supervised masking of real scans teaches the network what context predicts. COVAL-style synthetic insertion then provides true hidden background behind injected objects, so the model learns to output background rather than copy the occluder.

The identical-shortcut risk. Real training scans contain parked cars. The model may learn that cars along curbs are predictable, which would shrink residuals there.

We control this in two ways. We keep the context radius just larger than typical object footprints, so that an object's own points never appear in its context. We also let a DRÆM-style discriminator read the observed and inpainted views jointly, trained on synthetic insertions. The dependence of residual quality on context radius is itself a result we will report.

The per-point objectness score. For each point we combine the discriminator's output with ray consistency. A return that lands in front of the predicted background surface is evidence of an object. A predicted background surface that the laser should have hit but did not is evidence of occlusion.

Box fitting. We cluster high-scoring points. For each cluster, the observed front face gives a lower bound on extent. The predicted background along the occluded rays gives an upper bound on depth. We fit the 7-degree-of-freedom box by minimizing point-to-surface distance subject to these bounds. This replaces CPD's height-anchored prototype prior with a per-object physical constraint, and it works for classes with no complete exemplars.

Student training. Each pseudo-box's regression loss is down-weighted by the residual mass along its boundary. We then run standard self-training rounds with a CenterPoint student, following the LISO and CPD protocols.

Baselines

The naive reference is ground removal followed by density clustering and class-size filtering on the same single frames. The student detector, schedule and number of self-training rounds match ours exactly. Any gain we show must at minimum exceed this.

The strongest existing method is CPD, which reports 32.13% AP on Waymo Vehicle L2 and 90.85% / 81.01% AP on KITTI Easy / Moderate cars under Waymo-to-KITTI zero-shot transfer. We will run CPD's released protocol with the same student architecture and training budget. We will add CPD's multi-frame aggregation as an input option to our own pipeline, so the comparison does not reduce to single-frame versus multi-frame. To count as beating CPD, we must match its vehicle AP within noise across three seeds and exceed it on cyclists and never-moving classes. We found no reported CPD numbers for those classes on Lacuna, so we will generate them ourselves.

The motion baseline is LISO. It reports 0.448 [email protected] on Argoverse 2 and 0.350 / 0.322 AP for moving / still objects on Waymo. We run it with an identical student.

The uncertainty baseline is UA3D's dual-head weighting, which reports +6.9% BEV AP on nuScenes over its unsupervised base. We apply it on top of our pseudo-boxes, in place of our residual weighting.

Three ablations isolate the key ingredients:

  • Remove the inpainter and score clusters by density alone. This isolates the objectness signal.
  • Remove the depth bound. This isolates the truncation fix.
  • Remove the synthetic counterfactual training. This tests whether masked self-supervision alone suffers the identical shortcut.

Why not just…?

Why not just use motion and let the student generalize to static objects? LISO's still-object AP (0.322) is close to its moving-object AP (0.350). For vehicles, this alternative is a real threat, and we expect near-parity there. It cannot help with classes that are never seen moving, because the teacher emits zero labels for them. On nuScenes barriers and traffic cones, and on Waymo signs, motion-only training should score near zero. We will report exactly that split, so the claim stands or falls on those classes.

Why not just cluster non-ground points and use size priors, as CPD does? This is the strongest alternative, and it already avoids the motion dependence. Its limits are twofold. It cannot tell a cone from a bollard-shaped piece of wall or a bush without hand-set rules. And its prototype regularization needs complete exemplars, which fail for rare classes. The density-only ablation and the per-class comparison will show whether context residuals actually separate objects from clutter better. If they do not beat CPD on rare and never-moving classes, the proposal fails.

Why not just use free-space differencing across repeated drives of the same route? When revisit data exists, a location that is empty on another pass reveals movable objects cheaply. We will run this baseline on the Argoverse 2 and Waymo logs that have overlapping traversals. It cannot find permanently installed objects such as signs and barriers, and many fleets lack revisits. The inpainter's advantage should therefore concentrate there. If differencing matches us wherever revisits exist, we will say so.

Why not just click points with weak supervision? ViT-WSS3D lifts KITTI mAP from 24.4% to 48.3% using 2% box labels plus clicks on the rest. Clicks still scale with every object in every frame, and annotators click only known classes. We treat clicks as complementary rather than competing, and we report our zero-click result next to theirs.

Proposed Experiments

The primary benchmark is nuScenes, which has the never-moving barrier and traffic-cone classes. We report per-class 3D and BEV AP under unsupervised training, against all baselines. We found no reported unsupervised result for these classes on Lacuna.

Waymo tests generality. We report Vehicle L2 AP against CPD's 32.13%, and Cyclist and Sign AP. We also stratify results by moving versus stationary, following LISO's split, and by visibility or truncation bins. The depth-bound ablation is judged by box-size error on the most truncated bin.

KITTI zero-shot transfer from Waymo tests robustness to the sensor change, against CPD's 81.01% Moderate car AP.

STU tests open-world transfer. We convert our boxes to point masks and compare with COVAL's 89.00% AP and 57.95 panoptic quality. We also report box-level recall on the anomalies, for which we found no reported result on Lacuna.

The decisive checks are three:

  • Never-moving-class AP must be well above both LISO and the clustering baseline.
  • Vehicle AP must stay within noise of CPD.
  • Residual-based weighting must beat UA3D weighting on identical pseudo-boxes.

A plot of residual separability against range and object height will show whether the far-range and low-profile failures reported for COVAL and MOVES carry over. If they do, that limits the method's reach to near and mid range, and we will say so.