Laser scanners on cars only see the side of an object that faces them. To learn what the hidden side looks like, a shape-completion network has to borrow that knowledge from somewhere. Today it comes from one of two places. The first is clean synthetic CAD models, which carry strong shape priors but transfer poorly to real sensors. The second is real scans stitched together across time, which match the real sensor but give no labels once an object is far away or heavily blocked. We propose to stop choosing between them. For every laser ray that passes through a detected object, several independent "voters" each propose where the ray exits the hidden back surface: a network trained on synthetic data, the time-stitched real scan, a mirror image under the object's left–right symmetry, and a network that is invariant to rotation and scale. We combine their proposals with a generalized median mechanism, a voting rule from facility-location theory that takes the median of the voters' reports together with fixed "phantom" votes placed independently of those reports. Here the phantoms come from physics and category statistics: the exit point cannot lie in front of the observed hit, and it cannot lie beyond the object's bounding box. The fused depths become training labels for real-world completion.
The rule does two jobs. Because depth along a single ray is one-dimensional, the classical guarantee of median mechanisms applies exactly: no single voter, however wrong, can drag the label beyond the neighboring order statistic of the other votes and phantoms. This gives bounded influence without tuning any weights. The same structure also yields a measurement of synthetic reliance that the 3D literature lacks. For each ray we can ask whether the synthetic voter was pivotal, meaning that removing or perturbing it would change the label. The fraction of pivotal rays, stratified by range and occlusion, is a direct and interpretable measure of how much a real-world completion system depends on synthetic geometry.
Core Claim
Per-ray generalized-median fusion of synthetic, temporal, symmetric and equivariant depth proposals, with phantoms fixed by line-of-sight geometry, produces real-world supervision that extends below the sparsity floor of temporal aggregation. It stays robust when any one source is badly wrong. On distant and heavily occluded KITTI vehicles, where temporal ground truth barely exists, networks trained on these labels should beat both synthetic-only training and synthetic-pretraining-then-real-fine-tuning. The pivotality statistic that falls out of the mechanism should predict where synthetic data can and cannot substitute for real scans.
Where This Sits
The three directions play different roles. Geometric incompleteness is the application. Mechanism design supplies the aggregation rule. Synthetic-data evaluation supplies the measurement question.
On the geometry side, two lines of work motivate the gap from opposite ends. PCN: Point Completion Network set the standard coarse-to-fine encoder–decoder trained on synthetic ShapeNet. Learning Generalizable Shape Completion with SIM(3) Equivariance showed that such models memorize canonical poses. Its equivariant network narrows the gap zero-shot, yet it is still trained purely on synthetic shapes and never asks whether those shapes are right for real vehicles. At the other end, Self-supervised Point Cloud Completion on Real Traffic Scenes via Scene-concerned Bottom-up Mechanism (TraPCC) discards synthetic data. It builds a pool of real ground truth by stitching consecutive LiDAR frames of tracked vehicles and adding a symmetry mirror, then trains with a one-sided Chamfer loss. Its authors document the resulting sparsity floor: distant or occluded vehicles yield no usable ground truth. We reuse both systems as voters rather than competing with either. What we add is a principled way to combine them ray by ray, falling back on whichever sources remain informative.
Unsupervised Point Cloud Pre-training via Occlusion Completion (OcCo) supplies a method we reuse. It renders line-of-sight partial scans from synthetic meshes with z-buffering, and we use that renderer, via hansen7/OcCo, to build controlled experiments where complete geometry is known.
Flexible-weighted Chamfer Distance constrains the design. It shows that the balance between precision and coverage in point-set losses needs hand-tuned weights that vary by dataset, and that pushing coverage too hard erases thin structures. Our labels act per ray and do not rely on such weights. We still test FCD as a competing fix.
On the mechanism side, Mechanism Design for Facility Location Problems: A Survey and The Capacity Constrained Facility Location Problem establish that medians with report-independent phantoms are the canonical manipulation-proof rules on a line. The second of these proves they are the only such deterministic rules under capacity constraints. The same survey records the main obstacle: in two or more dimensions these guarantees largely disappear. Our key observation is that line-of-sight sensing restores a one-dimensional ordering, because each ray is a line. Facility Location Games with Scaling Effects shows that phantoms can be derived from known structure of the environment rather than chosen arbitrarily. We follow that spirit by deriving phantoms from sensor geometry and from category statistics computed on synthetic CAD models.
Augmented Normalization: Differentiating the Generalized Geometric Median matters as a contrast. It made a smoothed median trainable inside image networks and found that the pure median ($\phi=1$) hurt accuracy by 2–3% on CIFAR-10 because its gradients are sparse. We sidestep that negative result: our median runs offline, when labels are built, and is never backpropagated through. That is exactly where exactness helps rather than hurts.
The evaluation question comes from On the Equivalency, Substitutability, and Flexibility of Synthetic Data. For multi-person tracking it measured a roughly 30:1 exchange rate of synthetic to real frames and found that 60–80% of real data could be replaced by synthetic data. No such number exists for 3D completion. Is this Model for Real? supplies the methodological lesson that a model's authenticity should be judged against real structural noise. Our pivotality analysis applies that lesson to geometry.
The Mechanism
Consider a ray $r$ from the sensor that hits a detected object at observed depth $z_r$. Each of $m$ voters reports an exit depth $d_{r,1},\dots,d_{r,m}$. A voter that has no opinion on this ray, such as temporal aggregation with no frames, is replaced by a designated phantom rather than dropped, so that the rule always sees the same number of inputs. We add $m+1$ phantoms $p_{r,1},\dots,p_{r,m+1}$. These are computed only from $z_r$, the detector's bounding box, and category-level quantiles of exit depth measured on ShapeNet shapes placed in the same box and viewed from the same angle. The label is
$$ \hat d_r = \operatorname{median}\{d_{r,1},\dots,d_{r,m},\,p_{r,1},\dots,p_{r,m+1}\}. $$
The phantoms do not depend on any report. Changing a single voter's report can therefore move $\hat d_r$ only between the two order statistics of the remaining $2m$ values that bracket it. This is the one-dimensional bounded-influence property inherited from median mechanisms. It also guarantees that the label lies between the observed hit and the far edge of the box whenever the extreme phantoms sit there.
The design also clarifies the two ways synthetic data can enter the fused label. As a voter, it is a synthetic-trained completer's report. As a phantom, it is a fixed category prior. We test both.
The mechanism has a known limitation. It labels one exit depth per ray, so it captures the silhouette-bounded back surface of roughly convex objects such as vehicles well. It cannot label concavities that a ray crosses more than once. For that reason we restrict the main claims to vehicles and use one-sided Chamfer loss on observed points as a complementary term.
Research Program
We first build the voter pool. This consists of SIMECO and AdaPoinTr trained on PCN synthetic data, TraPCC-style temporal stitching, and the symmetry mirror, all applied to KITTI and nuScenes vehicles. For each ray through each detected object we compute the fused label. We then train a completion network on real scans only. The backbone matches SIMECO's so that comparisons isolate the choice of supervision. The loss has two parts: an L1 loss between predicted and fused exit depths along each ray, and TraPCC's one-sided Chamfer loss on visible points.
Next comes the analysis. For every ray we record which input is the median, whether the synthetic voter is pivotal, and how far the label would move if that voter were removed. We aggregate these by distance to the sensor, by the fraction of the object that is visible, and by vehicle subtype. Finally, we measure an exchange rate. We train with a varying fraction of real objects that have temporal ground truth, together with a varying number of synthetic shapes. The equivalency ratio is the number of synthetic shapes needed to reach the accuracy gained from one real stitched vehicle, following the definition of On the Equivalency, Substitutability, and Flexibility of Synthetic Data. We report it separately for near and far objects.
Baselines
The naive reference point is SIMECO trained only on PCN synthetic data and applied zero-shot. Its paper reports a 17% reduction in Minimal Matching Distance on KITTI relative to earlier completers (Learning Generalizable Shape Completion with SIM(3) Equivariance). We rerun it with the released recipe on our splits and do not rely on that relative figure.
The strongest existing real-world method is TraPCC, which reports a mean Chamfer distance of 0.0973 on KITTI (Self-supervised Point Cloud Completion on Real Traffic Scenes via Scene-concerned Bottom-up Mechanism). That paper also reports that its completions raise SECOND detector precision on "Hard" cases from 65.91% to 65.98%. We will reproduce TraPCC on identical splits and use the same backbone family, training epochs and tuning budget as our method. The main comparison is on the far and heavily occluded strata, where TraPCC's ground-truth pool is thin. Beating it means lower Chamfer and depth error on those strata while staying within noise of TraPCC on near vehicles.
The most dangerous practical baseline is synthetic pretraining followed by TraPCC fine-tuning: SIMECO initialized on PCN, then trained on the real ground-truth pool. We give it the same hyperparameter search budget as our method.
Three fusion ablations isolate the key ingredient by keeping the voters fixed and changing only the aggregation rule. The first is an unweighted mean of the voters. The second is a confidence-weighted mean, with weights tuned on a validation split. The third is a plain median without phantoms. If the phantom median does not beat the tuned weighted mean when one source is corrupted, the mechanism's contribution is not real. Separately, FCD applied to the TraPCC baseline tests whether reweighting the loss alone achieves the same effect.
Why not just…?
Why not just pretrain on synthetic data and fine-tune on TraPCC's real ground truth? This is the real threat. It may match us on near vehicles. The difference should appear only where temporal ground truth is missing. There, fine-tuning provides no real signal and the network falls back on its synthetic prior, whereas fused labels still combine symmetry, the equivariant model and physical phantoms. The range-stratified comparison decides this directly. If fine-tuning matches us on far vehicles, the fused labels add nothing and we will report that.
Why not just average the sources, or learn confidence weights? Averages have unbounded sensitivity. A single stitching failure, such as a misaligned frame or a passing truck merged into the car, shifts the label in proportion to its error. Learned weights have to be tuned per dataset, the same weakness that Flexible-weighted Chamfer Distance concedes for its loss weights. The corruption experiment below measures label error as a function of how badly one voter is corrupted. The phantom median should stay flat while the averages grow.
Why not just use heavy augmentation, or FCD, on the synthetic model? Learning Generalizable Shape Completion with SIM(3) Equivariance already found that augmentation does not remove alignment bias. Neither augmentation nor FCD changes what the supervision is: both keep training on synthetic shapes, which cannot model real vehicle distributions or the dropout patterns of LiDAR beam rings. We include FCD as a baseline and augmentation inside the naive reference, so the answer is measured rather than assumed.
Why not a smoothed, trainable median as in AugNorm? Labels are built offline, so gradients are unnecessary. Smoothing toward the mean would give up the exact bounded-influence property. As a check, we will run $\phi=1.5$ fusion as an ablation.
Proposed Experiments
The first experiment is controlled and runs on OmniObject3D real scans and PCN meshes, where complete geometry is known. We render line-of-sight partial scans with the OcCo pipeline and simulate each voter. We then corrupt one voter with growing error, including systematic bias, random outliers and total dropout, and measure fused-label error against the true exit depth. The prediction from the bounded-influence property is that phantom-median error plateaus at the spread of the remaining order statistics, while mean-based fusion grows linearly with the corruption.
The second experiment tests below the sparsity floor. We hold out densely stitched KITTI vehicles, which serve as reliable ground truth, and artificially thin them by dropping frames and cropping angular coverage. This lets us evaluate distant-like and occluded-like inputs against known complete shapes. We report Chamfer distance, F-score and per-ray depth error, stratified by the number of frames retained. Our method should beat the fine-tuned baseline most clearly in the lowest-frame bins.
The third experiment is downstream. Following TraPCC's protocol, we feed completed vehicles to SECOND on KITTI "Hard" cases and to a nuScenes-trained model tested on KITTI, and report detection precision.
The fourth experiment evaluates reliance. We report the pivotality rate of the synthetic voter by distance and visibility, the equivalency ratio for near and far strata, and whether pivotality predicts where adding synthetic shapes improves accuracy. The decisive check is a correlation across strata: high pivotality should go with a low exchange rate, meaning synthetic data is doing work that real data does not replace. If the two are uncorrelated, pivotality is not a useful measure of reliance, and we will say so.
References
- PCN: Point Completion Network
- Learning Generalizable Shape Completion with SIM(3) Equivariance
- Self-Supervised Point Cloud Completion on Real Traffic Scenes Via Scene-Concerned Bottom-Up Mechanism
- Unsupervised Point Cloud Pre-training via Occlusion Completion
- Flexible-weighted Chamfer Distance: Enhanced Objective Function for Point Cloud Completion
- Mechanism Design for Facility Location Problems: A Survey
- The Capacity Constrained Facility Location Problem
- Facility Location Games with Scaling Effects
- Augmented Normalization: Differentiating the Generalized Geometric Median
- On the Equivalency, Substitutability, and Flexibility of Synthetic Data
- Is this Model for Real? Simulating Data to Reveal the Proximity of a Model to Reality