Manufacturing Far-Range 3D Labels from Near-Range Objects

Training a 3D object detector for self-driving cars requires boxes drawn around objects in LiDAR point clouds. LiDAR is a laser scanner that returns a sparse set of 3D points. Each box takes a human about two minutes, and recent "label-free" pipelines replace the human with 2D vision foundation models plus geometric box fitting. These pipelines share a blind spot: they work on nearby objects, which return hundreds of laser points, and break on distant or occluded ones, which return a handful. The authors say so directly. The point-wrapping loss in General Geometry-Aware Weakly Supervised 3D Object Detection "lacks sufficient surface points" at range. The surface reconstruction in OpenBox fails to converge on sparse returns.

We propose to treat this as a paired-data scarcity problem and borrow the remedy from music generation. There, systems such as MuseCoco and TeleMelody never collect scarce text–music pairs at scale. They pick an intermediate representation that rules can extract from abundant unpaired data, which gives them unlimited training pairs for free. Our version works as follows. Nearby objects can be auto-labelled reliably. We re-scan each one, in simulation, as the actual sensor would see it at 40, 60, or 80 meters behind realistic occluders. That produces unlimited pairs of (sparse far-range observation, trustworthy box) without any human label. A small "sparse-to-box" model trained on these pairs then labels the far-range objects that existing auto-labellers cannot fit.

Core Claim

The missing ingredient at long range is not more labels. It is correctly paired examples of what a sparse object looks like together with its true box. Those pairs can be manufactured from the near-range objects that existing pipelines already label well. The manufacturing must respect the sensor's physics: a distant car loses whole laser rings and the side facing away from the sensor, not a random subset of points.

The bridge representation is a dense object template. It consists of an object's points aggregated over several frames while it is close to the sensor, together with its fitted 7-degree-of-freedom box (center, size, heading) and the class assigned by a 2D foundation model. Extracting templates is rule-based and needs no annotation, much as MuseCoco extracts tempo and key from raw MIDI files deterministically. A learned stage then maps a sparse far-range observation, plus the camera's 2D box when one is available, back to the template's box.

If this works, far-range average precision of auto-labels and of the detectors trained on them should rise sharply. Near-range accuracy should stay unchanged. The gain should vanish when the re-scanning ignores beam geometry.

Research Program

Template harvesting. We start from an existing label-free pipeline: OpenBox-style instance discovery with Grounding DINO and SAM2, which lifts 2D masks into the point cloud. We keep only objects whose boxes we trust. These are rigid objects observed within roughly 30 meters across many frames, where multi-frame aggregation and box fitting are known to be stable. For pedestrians, which deform and therefore smear when frames are stacked, we keep single-frame templates from the closest observations only. That restriction keeps the bridge's information bottleneck explicit rather than hidden, the same issue TeleMelody faces with its hand-derived templates.

Physics-faithful re-scanning. For each template we choose a target distance and bearing sampled from where objects actually occur in the dataset. We place the template there inside a real scene and cast the rays of the real sensor. For nuScenes that is the 32-beam pattern; for Waymo and KITTI, the respective sensors. Points that real scene geometry would occlude are removed, and sensor dropout is applied at the rate measured on that sensor. The output is a sparse point set with an exact box inherited from the template. In the same scene we can also render the template's projected 2D box, so the camera-side input for training costs nothing.

This differs from the proxy injection in ALPI. ALPI synthesizes proxy objects with known boxes from class size priors and injects them into scenes to bootstrap a frustum network. We inject real harvested object surfaces. We also re-render them specifically to match far-range sparsity, which is the regime where every surveyed method fails.

Sparse-to-box model. A lightweight point-set network takes three inputs: the sparse observation, the 2D box when the camera sees the object, and the class. It regresses the full 3D box. We supervise it with the inherited template box and a depth-normalized 2D projection term in the style of ALPI's normalized loss.

When the camera input is absent, the model runs on LiDAR alone. This matters at night, since nuScenes reports vision-based detection degrading by 55–58%, against 36% for LiDAR.

The design also takes a position on the decoupling debate between Motal and DFU3D. Motal argues for separating the modalities by task. DFU3D argues for fusing them at the data level. We assign the camera to what and where in the image, and LiDAR to how far. Unlike DFU3D, we never back-project monocular depth into fake points, because DFU3D itself reports that those pseudo-points cause scale drift and edge stretching.

Labelling and detector training. Far-range and occluded instances found by 2D foundation models are labelled by the sparse-to-box model. Near-range instances keep the geometric-fitting labels. We train a standard CenterPoint detector on the union of both sets. There is no iterative self-training loop, which avoids the multi-round cost that ALPI incurs.

Baselines

The naive reference point is the upstream auto-labeller used alone: OpenBox-style 2D-mask lifting with geometric box fitting at every range. We build it ourselves so that the instance discovery is identical to ours, and train the same CenterPoint detector on its labels. On the Waymo Open Dataset, OpenBox reports 70.49% 3D average precision for vehicles at IoU 0.5 with no human 3D labels. We will first reproduce a figure near this value before any comparison. Beating this baseline means a clear gain in far-range average precision (beyond 50 m) with no loss inside 30 m. We found no reported distance-binned result on Lacuna for OpenBox, so we will report those bins ourselves.

The strongest unsupervised competitor on nuScenes is DFU3D. It densifies LiDAR with monocular-depth pseudo-points and reports 28.4% class-aware mAP, 15.0% class-agnostic 3D average precision, and 15.3% mAP on cyclists over the 0–80 m range. It is the most direct alternative answer to sparsity, so it is the decisive comparison. We will run it, or use its published numbers when protocols match, under the same 0–80 m evaluation and the same detector backbone. To claim a win we must exceed 28.4% mAP and show that the gain concentrates in far range.

On KITTI, the strongest method that uses 2D labels is ALPI. It reports 79.43% moderate-difficulty Car mAP with PointRCNN. That number comes from human 2D boxes, and ALPI reaches about 70% with Faster R-CNN detections. We compare in both settings, holding the 2D source fixed. We also include Motal on Waymo. Motal reports gains of 21.56 mAP L1 over CPD but gives no per-range numbers, so we will measure them.

The key ablation keeps everything identical but replaces physics-faithful re-scanning with uniform random point dropping to the same point count. A second ablation swaps real harvested templates for ALPI-style synthetic size-prior proxies. Together they separate the contribution of "real surfaces" from that of "realistic sparsity". Every variant gets the same number of synthetic pairs, the same training steps, and the same hyperparameter search budget.

Why not just…?

Why not just augment the detector with distance-based downsampling on existing pseudo-labels? This is the real threat, and it is exactly our first ablation. If uniform downsampling matches physics-faithful re-scanning, the core claim fails, and the contribution reduces to "train on harvested near-range labels with augmentation". We expect it not to match. Uniform dropping keeps points on the far side of the object and keeps every laser ring. Real distance removes whole rings and back faces, and that drives the depth and heading drift GGA describes. The far-range heading error metric will settle the question.

Why not just densify with monocular depth? DFU3D does this and is our main competitor. Its own analysis reports metric scale drift and edge stretching, which grow precisely at range. It also requires heavy filtering anchored to real LiDAR points, and at range those anchors are scarce. Our method adds no fake points and also runs without the camera, so a nighttime split of nuScenes tests the difference.

Why not just spend a little human effort on bird's-eye-view center clicks? WS3D reaches 75.10% on KITTI moderate Car at 2.5 seconds per click. It still needs a fully labelled seed, assumes flat ground, and must be redone for every new domain and class. We treat clicks as complementary rather than competing. In one experiment we spend an equal human-time budget on clicks, either for far-range objects alone or on top of our labels, to see whether manufactured pairs make clicks unnecessary or merely cheaper.

Why not just rely on LiDAR motion cues? LISO finds movers through scene flow, the per-point motion between consecutive scans. It cannot assign classes, and it inherits the same far-range sparsity during clustering. Motion could serve as a further source of templates, but it does not address the range failure.

Proposed Experiments

We run on the Waymo Open Dataset (vehicles, pedestrians, cyclists; Level 1 and Level 2 AP and APH), nuScenes (0–80 m mAP, AP_BEV, AP_3D, per-class), and KITTI (Easy/Moderate/Hard). Every metric is reported in distance bins of 0–30, 30–50, and beyond 50 m, and in bins of points per object. The bins are where the claim lives.

Before any detector training, we measure auto-label quality directly against held-out human boxes, as a function of distance. This checks whether the sparse-to-box model beats geometric fitting on exactly the objects where fitting fails.

The first decisive check is the re-scanning ablation described above.

The second tests whether templates harvested in one city transfer to another. The Waymo Open Dataset already documents an 8.0 APH drop from suburban to dense urban San Francisco. We harvest templates in suburban scenes and evaluate far-range labelling in San Francisco.

The third tests pedestrians and cyclists separately. Single-frame templates may carry too little shape information, and Motal and DFU3D both report weak small-object performance. If gains appear only for rigid vehicles, we will say so and treat deformable objects as the bridge's stated limit.